2021-03-09T08:49:45.093 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-09T08:49:45.095 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} --verbose --archive /home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949100 --name kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi --owner scheduled_kchai@teuthology -- /home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949100/orig.config.yaml 2021-03-09T08:49:45.128 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-03-09T08:49:45.183 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949100 branch: wip-kefu-testing-2021-03-09-1428 description: rados/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} email: tchaikov@gmail.com first_in_suite: false job_id: '5949100' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-kefu-testing-2021-03-09-1428 ceph: conf: client: debug ms: 1 global: mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd shutdown pgref assert: true log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 cephadm: cephadm_mode: root install: ceph: sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 workunit: branch: wip-kefu-testing-2021-03-09-1428 sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 owner: scheduled_kchai@teuthology priority: 60 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.realm.zone.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2021-03-09-1428 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 targets: smithi005.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5imz6eFl6OjdUdzuCQGzroe92h0RS7f0Or141dq61G8bh33AmzP1w+2XykdFnV/6KAV5zORA20epSl8WKoLHze9MWy3+QyrqrLSYsOoOxwNcjpsr0ayt2AqAqpYtpQiUWmK48GZDzl4si+IAJ3qDd4WlgiCd25CLfp2ZKctYKRTuXLIOtcmzcFOudwOFHhmWLF+d+Zw8AN4stVsaK7o8BCXfliGbeNMuxslx3kcQ3sbjzPyMHriver1R6MJQ7IiLdfx1ao4j+lz0mP3BjjcSKbO83kHEkTqrwBG1Rk0qJ94tqPJnYwqGKBJd/oiXp4y0Ebl105zNiOf8ZP6dWVC8fJ2i5H1R3S+mPNxhDAN6S0lH0/UgtJ0eRWoMrzs5KVBPseRia3R6m7sb5WVrtKgB2upzTqqCBZY2If3aJao+PAxCBbywa6IjVmbY+R7Ni6HJ8OA5nAuYyxczQT/4cZyOkHMexl2bzgQmyC84N333wMZuGZET0+43RRFt9sSlTWss= smithi146.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjh1xdEq0UlWDcOjLO0zuv0sSEWyEHQ40609AUTn+yJaQabMNBppSsxY3EilBaU0bsJR+1vvI/zVjwHxSVkI17jHLEBffiPFWF8Zk0q5BnykNpZFW4WQXY7Kzan8+FUeW7Ouq/FoPFuEGg5yojgeJLzlLdY4ba62VxjnCW5r11afjcW5JG3yJHezu6ukf1dQ0wATASmf1aAQYS23zVa1dWRT9n0zkDbiJZY5Km9TaKmV2Wq4TkPYdWQKzJ4Twc2bRqUcUo07CKxB3aQ5l385R2Qwe9wYVQpqmSr4EXrkrgVzwuqsDMs7PO77Y7vw//bjUO0JfydAuMuMmHiZx1uj8AQL2gnp8o0MffYgsT4JKODG3jan9Kr9e/AsQiMZtZb3RVkVK/n72er6Sww4HUkGZVsn+D9h8PNbytero37DLSHzqqD/sQS271fbn9a9R6yz+YqTSfXJQfYSmgz6Mtcda6OyLsoZ7AKMlNjqdlOdsPEw4vyXqJKyn0Vv3nWPX7P9k= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology_branch: master teuthology_sha1: 3082387bba74fcd24c9700593d10418152d53c97 timestamp: 2021-03-09_08:22:40 tube: smithi user: kchai verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.25973 2021-03-09T08:49:45.183 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa; will attempt to use it 2021-03-09T08:49:45.189 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa/tasks 2021-03-09T08:49:45.190 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-03-09T08:49:45.192 INFO:teuthology.task.internal:Checking packages... 2021-03-09T08:49:45.239 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '89fb622b10ca2e58fe0c913a8956377a886e2ab4' 2021-03-09T08:49:45.240 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-09T08:49:45.240 INFO:teuthology.packaging:ref: None 2021-03-09T08:49:45.241 INFO:teuthology.packaging:tag: None 2021-03-09T08:49:45.241 INFO:teuthology.packaging:branch: wip-kefu-testing-2021-03-09-1428 2021-03-09T08:49:45.242 INFO:teuthology.packaging:sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:49:45.242 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-kefu-testing-2021-03-09-1428 2021-03-09T08:49:45.399 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-1742-g89fb622b-1focal 2021-03-09T08:49:45.400 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-03-09T08:49:45.411 INFO:teuthology.task.internal:no buildpackages task found 2021-03-09T08:49:45.411 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-03-09T08:49:45.443 INFO:teuthology.task.internal:Saving configuration 2021-03-09T08:49:45.455 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-03-09T08:49:45.469 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-09T08:49:45.488 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-09 08:44:00.927143', 'description': '/home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949100', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEsynB/TKk8dM0+AzlynUTMNrGvkg2OxO/OhEX2tAIfh', 'os_version': '20.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi005.front.sepia.ceph.com'} 2021-03-09T08:49:45.506 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-09 08:44:00.927122', 'description': '/home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949100', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDouh1+y6/5AJLGxw0vmtDfs8xnMToO9bBTf3f/QN1K5', 'os_version': '20.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi146.front.sepia.ceph.com'} 2021-03-09T08:49:45.506 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-03-09T08:49:45.516 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.realm.zone.a', 'node-exporter.a', 'alertmanager.a'] 2021-03-09T08:49:45.517 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2021-03-09T08:49:45.517 INFO:teuthology.run_tasks:Running task console_log... 2021-03-09T08:49:45.642 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6b944de9d8>, signals=[15]) 2021-03-09T08:49:45.643 INFO:teuthology.run_tasks:Running task internal.connect... 2021-03-09T08:49:45.731 INFO:teuthology.task.internal:Opening connections... 2021-03-09T08:49:45.732 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2021-03-09T08:49:45.734 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T08:49:45.795 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2021-03-09T08:49:45.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T08:49:45.857 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-03-09T08:49:45.867 DEBUG:teuthology.orchestra.run.smithi005:> uname -m 2021-03-09T08:49:45.875 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2021-03-09T08:49:45.875 DEBUG:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2021-03-09T08:49:45.923 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="Ubuntu" 2021-03-09T08:49:45.923 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-03-09T08:49:45.923 INFO:teuthology.orchestra.run.smithi005.stdout:ID=ubuntu 2021-03-09T08:49:45.923 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE=debian 2021-03-09T08:49:45.924 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-03-09T08:49:45.924 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="20.04" 2021-03-09T08:49:45.924 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-09T08:49:45.924 INFO:teuthology.orchestra.run.smithi005.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-09T08:49:45.925 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-09T08:49:45.925 INFO:teuthology.orchestra.run.smithi005.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-09T08:49:45.925 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_CODENAME=focal 2021-03-09T08:49:45.925 INFO:teuthology.orchestra.run.smithi005.stdout:UBUNTU_CODENAME=focal 2021-03-09T08:49:45.926 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2021-03-09T08:49:45.942 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2021-03-09T08:49:45.949 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2021-03-09T08:49:45.949 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2021-03-09T08:49:46.002 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="Ubuntu" 2021-03-09T08:49:46.003 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-03-09T08:49:46.003 INFO:teuthology.orchestra.run.smithi146.stdout:ID=ubuntu 2021-03-09T08:49:46.003 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE=debian 2021-03-09T08:49:46.004 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-03-09T08:49:46.004 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="20.04" 2021-03-09T08:49:46.004 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-09T08:49:46.005 INFO:teuthology.orchestra.run.smithi146.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-09T08:49:46.005 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-09T08:49:46.005 INFO:teuthology.orchestra.run.smithi146.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-09T08:49:46.005 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_CODENAME=focal 2021-03-09T08:49:46.006 INFO:teuthology.orchestra.run.smithi146.stdout:UBUNTU_CODENAME=focal 2021-03-09T08:49:46.007 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2021-03-09T08:49:46.022 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-03-09T08:49:46.036 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-03-09T08:49:46.093 INFO:teuthology.task.internal:Checking for old test directory... 2021-03-09T08:49:46.093 DEBUG:teuthology.orchestra.run.smithi005:> test '!' -e /home/ubuntu/cephtest 2021-03-09T08:49:46.096 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2021-03-09T08:49:46.103 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-03-09T08:49:46.126 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-03-09T08:49:46.126 DEBUG:teuthology.orchestra.run.smithi005:> test -z $(ls -A /var/lib/ceph) 2021-03-09T08:49:46.129 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2021-03-09T08:49:46.141 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-03-09T08:49:46.218 INFO:teuthology.run_tasks:Running task kernel... 2021-03-09T08:49:46.239 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'ceph.rgw.realm.zone.a': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.a': {'kdb': True, 'sha1': 'distro'}, 'alertmanager.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'prometheus.a': {'kdb': True, 'sha1': 'distro'}, 'grafana.a': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.b': {'kdb': True, 'sha1': 'distro'}, 'ceph.iscsi.iscsi.a': {'kdb': True, 'sha1': 'distro'}} 2021-03-09T08:49:46.239 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-03-09T08:49:46.240 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2021-03-09T08:49:46.247 INFO:teuthology.orchestra.run.smithi005.stdout:5.4.0-66-generic 2021-03-09T08:49:46.248 INFO:teuthology.task.kernel:Running kernel on smithi005: 5.4.0-66-generic 2021-03-09T08:49:46.248 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get clean 2021-03-09T08:49:46.347 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2021-03-09T08:49:46.734 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2021-03-09T08:49:46.734 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [109 kB] 2021-03-09T08:49:46.788 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2021-03-09T08:49:47.117 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB] 2021-03-09T08:49:47.178 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [198 kB] 2021-03-09T08:49:47.325 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [842 kB] 2021-03-09T08:49:47.369 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [524 kB] 2021-03-09T08:49:47.494 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [113 kB] 2021-03-09T08:49:47.512 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [7,172 B] 2021-03-09T08:49:47.513 INFO:teuthology.orchestra.run.smithi005.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [542 kB] 2021-03-09T08:49:47.564 INFO:teuthology.orchestra.run.smithi005.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [428 kB] 2021-03-09T08:49:47.602 INFO:teuthology.orchestra.run.smithi005.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [434 kB] 2021-03-09T08:49:47.603 INFO:teuthology.orchestra.run.smithi005.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [78.1 kB] 2021-03-09T08:49:47.607 INFO:teuthology.orchestra.run.smithi005.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [10.5 kB] 2021-03-09T08:49:47.626 INFO:teuthology.orchestra.run.smithi005.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [201 kB] 2021-03-09T08:49:47.639 INFO:teuthology.orchestra.run.smithi005.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [12.8 kB] 2021-03-09T08:49:47.640 INFO:teuthology.orchestra.run.smithi005.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [555 kB] 2021-03-09T08:49:47.693 INFO:teuthology.orchestra.run.smithi005.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [747 kB] 2021-03-09T08:49:47.734 INFO:teuthology.orchestra.run.smithi005.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [155 kB] 2021-03-09T08:49:47.741 INFO:teuthology.orchestra.run.smithi005.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [16.2 kB] 2021-03-09T08:49:50.858 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 5,188 kB in 2s (3,223 kB/s) 2021-03-09T08:49:51.978 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2021-03-09T08:49:52.001 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-09T08:49:52.073 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2021-03-09T08:49:52.230 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2021-03-09T08:49:52.230 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2021-03-09T08:49:52.308 INFO:teuthology.orchestra.run.smithi005.stderr:E: Unable to locate package linux-image-current-generic 2021-03-09T08:49:52.310 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-09T08:49:52.311 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-09T08:49:52.376 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2021-03-09T08:49:52.532 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2021-03-09T08:49:52.533 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2021-03-09T08:49:52.697 INFO:teuthology.orchestra.run.smithi005.stdout:linux-image-generic is already the newest version (5.4.0.66.69). 2021-03-09T08:49:52.698 INFO:teuthology.orchestra.run.smithi005.stdout:linux-image-generic set to manually installed. 2021-03-09T08:49:52.698 INFO:teuthology.orchestra.run.smithi005.stdout:0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded. 2021-03-09T08:49:52.699 DEBUG:teuthology.orchestra.run.smithi005:> dpkg -s linux-image-generic 2021-03-09T08:49:52.719 INFO:teuthology.orchestra.run.smithi005.stdout:Package: linux-image-generic 2021-03-09T08:49:52.720 INFO:teuthology.orchestra.run.smithi005.stdout:Status: install ok installed 2021-03-09T08:49:52.720 INFO:teuthology.orchestra.run.smithi005.stdout:Priority: optional 2021-03-09T08:49:52.720 INFO:teuthology.orchestra.run.smithi005.stdout:Section: kernel 2021-03-09T08:49:52.720 INFO:teuthology.orchestra.run.smithi005.stdout:Installed-Size: 18 2021-03-09T08:49:52.721 INFO:teuthology.orchestra.run.smithi005.stdout:Maintainer: Ubuntu Kernel Team 2021-03-09T08:49:52.721 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: amd64 2021-03-09T08:49:52.721 INFO:teuthology.orchestra.run.smithi005.stdout:Source: linux-meta 2021-03-09T08:49:52.721 INFO:teuthology.orchestra.run.smithi005.stdout:Version: 5.4.0.66.69 2021-03-09T08:49:52.722 INFO:teuthology.orchestra.run.smithi005.stdout:Provides: virtualbox-guest-modules (= 6.1.16-dfsg-6~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.6) 2021-03-09T08:49:52.722 INFO:teuthology.orchestra.run.smithi005.stdout:Depends: linux-image-5.4.0-66-generic, linux-modules-extra-5.4.0-66-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-09T08:49:52.722 INFO:teuthology.orchestra.run.smithi005.stdout:Recommends: thermald 2021-03-09T08:49:52.722 INFO:teuthology.orchestra.run.smithi005.stdout:Description: Generic Linux kernel image 2021-03-09T08:49:52.723 INFO:teuthology.orchestra.run.smithi005.stdout: This package will always depend on the latest generic kernel image 2021-03-09T08:49:52.723 INFO:teuthology.orchestra.run.smithi005.stdout: available. 2021-03-09T08:49:52.723 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-03-09T08:49:52.724 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2021-03-09T08:49:52.731 INFO:teuthology.orchestra.run.smithi146.stdout:5.4.0-66-generic 2021-03-09T08:49:52.731 INFO:teuthology.task.kernel:Running kernel on smithi146: 5.4.0-66-generic 2021-03-09T08:49:52.731 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get clean 2021-03-09T08:49:52.817 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get update 2021-03-09T08:49:52.968 INFO:teuthology.orchestra.run.smithi146.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [109 kB] 2021-03-09T08:49:53.079 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-03-09T08:49:53.159 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2021-03-09T08:49:53.236 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [198 kB] 2021-03-09T08:49:53.291 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [524 kB] 2021-03-09T08:49:53.316 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [113 kB] 2021-03-09T08:49:53.320 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [7,172 B] 2021-03-09T08:49:53.321 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [428 kB] 2021-03-09T08:49:53.329 INFO:teuthology.orchestra.run.smithi146.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [542 kB] 2021-03-09T08:49:53.338 INFO:teuthology.orchestra.run.smithi146.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [78.1 kB] 2021-03-09T08:49:53.340 INFO:teuthology.orchestra.run.smithi146.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [10.5 kB] 2021-03-09T08:49:53.493 INFO:teuthology.orchestra.run.smithi146.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB] 2021-03-09T08:49:53.619 INFO:teuthology.orchestra.run.smithi146.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [434 kB] 2021-03-09T08:49:53.846 INFO:teuthology.orchestra.run.smithi146.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [842 kB] 2021-03-09T08:49:53.955 INFO:teuthology.orchestra.run.smithi146.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [201 kB] 2021-03-09T08:49:53.975 INFO:teuthology.orchestra.run.smithi146.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [12.8 kB] 2021-03-09T08:49:53.976 INFO:teuthology.orchestra.run.smithi146.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [555 kB] 2021-03-09T08:49:54.013 INFO:teuthology.orchestra.run.smithi146.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [747 kB] 2021-03-09T08:49:54.050 INFO:teuthology.orchestra.run.smithi146.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [155 kB] 2021-03-09T08:49:54.057 INFO:teuthology.orchestra.run.smithi146.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [16.2 kB] 2021-03-09T08:49:56.986 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 5,188 kB in 1s (3,822 kB/s) 2021-03-09T08:49:58.098 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2021-03-09T08:49:58.121 DEBUG:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-09T08:49:58.191 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2021-03-09T08:49:58.347 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2021-03-09T08:49:58.348 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2021-03-09T08:49:58.422 INFO:teuthology.orchestra.run.smithi146.stderr:E: Unable to locate package linux-image-current-generic 2021-03-09T08:49:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-09T08:49:58.425 DEBUG:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-09T08:49:58.491 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2021-03-09T08:49:58.631 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2021-03-09T08:49:58.631 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2021-03-09T08:49:58.771 INFO:teuthology.orchestra.run.smithi146.stdout:linux-image-generic is already the newest version (5.4.0.66.69). 2021-03-09T08:49:58.772 INFO:teuthology.orchestra.run.smithi146.stdout:linux-image-generic set to manually installed. 2021-03-09T08:49:58.772 INFO:teuthology.orchestra.run.smithi146.stdout:0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded. 2021-03-09T08:49:58.773 DEBUG:teuthology.orchestra.run.smithi146:> dpkg -s linux-image-generic 2021-03-09T08:49:58.787 INFO:teuthology.orchestra.run.smithi146.stdout:Package: linux-image-generic 2021-03-09T08:49:58.788 INFO:teuthology.orchestra.run.smithi146.stdout:Status: install ok installed 2021-03-09T08:49:58.788 INFO:teuthology.orchestra.run.smithi146.stdout:Priority: optional 2021-03-09T08:49:58.788 INFO:teuthology.orchestra.run.smithi146.stdout:Section: kernel 2021-03-09T08:49:58.788 INFO:teuthology.orchestra.run.smithi146.stdout:Installed-Size: 18 2021-03-09T08:49:58.788 INFO:teuthology.orchestra.run.smithi146.stdout:Maintainer: Ubuntu Kernel Team 2021-03-09T08:49:58.789 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: amd64 2021-03-09T08:49:58.789 INFO:teuthology.orchestra.run.smithi146.stdout:Source: linux-meta 2021-03-09T08:49:58.789 INFO:teuthology.orchestra.run.smithi146.stdout:Version: 5.4.0.66.69 2021-03-09T08:49:58.789 INFO:teuthology.orchestra.run.smithi146.stdout:Provides: virtualbox-guest-modules (= 6.1.16-dfsg-6~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.6) 2021-03-09T08:49:58.789 INFO:teuthology.orchestra.run.smithi146.stdout:Depends: linux-image-5.4.0-66-generic, linux-modules-extra-5.4.0-66-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-09T08:49:58.790 INFO:teuthology.orchestra.run.smithi146.stdout:Recommends: thermald 2021-03-09T08:49:58.790 INFO:teuthology.orchestra.run.smithi146.stdout:Description: Generic Linux kernel image 2021-03-09T08:49:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: This package will always depend on the latest generic kernel image 2021-03-09T08:49:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: available. 2021-03-09T08:49:58.791 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-03-09T08:49:58.791 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-03-09T08:49:58.791 DEBUG:teuthology.orchestra.run.smithi005:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-09T08:49:58.812 INFO:teuthology.orchestra.run.smithi005.stdout:ttyS1 2021-03-09T08:49:58.846 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2021-03-09T08:49:58.847 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-09T08:49:58.870 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2021-03-09T08:49:58.894 INFO:teuthology.run_tasks:Running task internal.base... 2021-03-09T08:49:58.903 INFO:teuthology.task.internal:Creating test directory... 2021-03-09T08:49:58.905 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-09T08:49:58.908 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-09T08:49:58.942 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-03-09T08:49:58.953 INFO:teuthology.run_tasks:Running task internal.archive... 2021-03-09T08:49:59.000 INFO:teuthology.task.internal:Creating archive directory... 2021-03-09T08:49:59.001 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-09T08:49:59.004 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-09T08:49:59.015 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-03-09T08:49:59.095 INFO:teuthology.task.internal:Enabling coredump saving... 2021-03-09T08:49:59.096 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-09T08:49:59.098 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-09T08:49:59.124 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T08:49:59.125 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T08:49:59.135 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T08:49:59.135 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T08:49:59.137 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-03-09T08:49:59.147 INFO:teuthology.task.internal:Configuring sudo... 2021-03-09T08:49:59.148 DEBUG:teuthology.orchestra.run.smithi005:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-09T08:49:59.151 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-09T08:49:59.194 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-03-09T08:49:59.205 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-03-09T08:49:59.206 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-09T08:49:59.208 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-09T08:49:59.215 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-09T08:49:59.222 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-09T08:49:59.229 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2021-03-09T08:49:59.230 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-09T08:49:59.315 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-09T08:49:59.323 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-09T08:49:59.371 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2021-03-09T08:49:59.371 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-09T08:49:59.393 DEBUG:teuthology.orchestra.run.smithi005:> sudo service rsyslog restart 2021-03-09T08:49:59.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2021-03-09T08:49:59.447 INFO:teuthology.run_tasks:Running task internal.timer... 2021-03-09T08:49:59.456 INFO:teuthology.task.internal:Starting timer... 2021-03-09T08:49:59.457 INFO:teuthology.run_tasks:Running task pcp... 2021-03-09T08:49:59.515 INFO:teuthology.run_tasks:Running task selinux... 2021-03-09T08:49:59.550 DEBUG:teuthology.task.selinux:Excluding smithi005: OS 'ubuntu' does not support SELinux 2021-03-09T08:49:59.551 DEBUG:teuthology.task.selinux:Excluding smithi146: OS 'ubuntu' does not support SELinux 2021-03-09T08:49:59.551 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-03-09T08:49:59.552 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-03-09T08:49:59.552 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-03-09T08:49:59.552 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-03-09T08:49:59.600 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2021-03-09T08:49:59.612 INFO:teuthology.repo_utils:Fetching master from origin 2021-03-09T08:49:59.754 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-03-09T08:49:59.921 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-03-09T08:49:59.924 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi005.front.sepia.ceph.com,smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-03-09T08:50:00.772 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-03-09T08:50:02.477 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-03-09T08:50:05.011 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-03-09T08:50:05.061 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:05 +0000 (0:00:00.083) 0:00:00.083 ********* 2021-03-09T08:50:05.077 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:05 +0000 (0:00:00.016) 0:00:00.099 ********* 2021-03-09T08:50:07.270 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-03-09T08:50:07.272 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 20.04 on host smithi146.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi146.front.sepia.ceph.com] 2021-03-09T08:50:07.324 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:07 +0000 (0:00:02.247) 0:00:02.346 ********* 2021-03-09T08:50:07.400 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 20.04 on host smithi005.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi005.front.sepia.ceph.com] 2021-03-09T08:50:07.454 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:07 +0000 (0:00:00.129) 0:00:02.476 ********* 2021-03-09T08:50:07.617 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-03-09T08:50:07.619 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-09T08:50:07.639 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:07 +0000 (0:00:00.185) 0:00:02.661 ********* 2021-03-09T08:50:07.646 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-09T08:50:07.671 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:07 +0000 (0:00:00.031) 0:00:02.693 ********* 2021-03-09T08:50:08.321 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** changed: [smithi005.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-03-09T08:50:08.349 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:08 +0000 (0:00:00.677) 0:00:03.371 ********* 2021-03-09T08:50:08.366 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-03-09T08:50:08.391 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:08 +0000 (0:00:00.041) 0:00:03.412 ********* 2021-03-09T08:50:08.722 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-03-09T08:50:08.724 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.051088", "end": "2021-03-09 08:50:08.381978", "rc": 0, "start": "2021-03-09 08:50:08.330890", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.064593", "end": "2021-03-09 08:50:09.122691", "rc": 0, "start": "2021-03-09 08:50:09.058098", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-03-09T08:50:08.746 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:08 +0000 (0:00:00.355) 0:00:03.768 ********* 2021-03-09T08:50:08.769 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:08 +0000 (0:00:00.022) 0:00:03.791 ********* 2021-03-09T08:50:09.007 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-03-09T08:50:09.009 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi005.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:50:09.035 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:09 +0000 (0:00:00.266) 0:00:04.057 ********* 2021-03-09T08:50:09.056 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:09 +0000 (0:00:00.020) 0:00:04.078 ********* 2021-03-09T08:50:09.956 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-03-09T08:50:09.957 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279809.0892687-9092-60039863522793/source", "state": "file", "uid": 0} 2021-03-09T08:50:09.979 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:09 +0000 (0:00:00.923) 0:00:05.001 ********* 2021-03-09T08:50:09.989 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279809.0694697-9090-233878939967323/source", "state": "file", "uid": 0} 2021-03-09T08:50:10.025 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:10 +0000 (0:00:00.045) 0:00:05.047 ********* 2021-03-09T08:50:10.535 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-03-09T08:50:10.537 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-09T08:50:10.538 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-09T08:50:10.559 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-09T08:50:10.561 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-09T08:50:10.647 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-03-09T08:50:10.649 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:10.650 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-03-09T08:50:10.824 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:10 +0000 (0:00:00.798) 0:00:05.845 ********* 2021-03-09T08:50:10.844 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:10 +0000 (0:00:00.020) 0:00:05.865 ********* 2021-03-09T08:50:11.691 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi005.front.sepia.ceph.com] 2021-03-09T08:50:11.733 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] 2021-03-09T08:50:11.769 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-03-09T08:50:11.771 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:11 +0000 (0:00:00.925) 0:00:06.791 ********* 2021-03-09T08:50:11.820 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:11 +0000 (0:00:00.050) 0:00:06.842 ********* 2021-03-09T08:50:12.541 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-03-09T08:50:12.543 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.406301", "end": "2021-03-09 08:50:12.512125", "rc": 0, "start": "2021-03-09 08:50:12.105824", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.406304", "end": "2021-03-09 08:50:12.512128", "rc": 0, "start": "2021-03-09 08:50:12.105824", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} 2021-03-09T08:50:12.565 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.744) 0:00:07.587 ********* 2021-03-09T08:50:12.587 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.021) 0:00:07.608 ********* 2021-03-09T08:50:12.708 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-03-09T08:50:12.710 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013298", "end": "2021-03-09 08:50:12.383774", "failed_when_result": false, "rc": 0, "start": "2021-03-09 08:50:12.370476", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-09T08:50:12.734 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.147) 0:00:07.756 ********* 2021-03-09T08:50:12.743 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.008222", "end": "2021-03-09 08:50:13.125839", "failed_when_result": false, "rc": 0, "start": "2021-03-09 08:50:13.117617", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-09T08:50:12.768 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.034) 0:00:07.790 ********* 2021-03-09T08:50:12.807 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-03-09T08:50:12.809 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:12.839 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.071) 0:00:07.861 ********* 2021-03-09T08:50:12.857 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:12.882 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.042) 0:00:07.903 ********* 2021-03-09T08:50:12.919 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-03-09T08:50:12.921 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:12.945 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.063) 0:00:07.967 ********* 2021-03-09T08:50:12.953 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:12.983 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:12 +0000 (0:00:00.037) 0:00:08.004 ********* 2021-03-09T08:50:13.032 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:13.058 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.075) 0:00:08.079 ********* 2021-03-09T08:50:13.065 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:13.094 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.035) 0:00:08.115 ********* 2021-03-09T08:50:13.133 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-03-09T08:50:13.135 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:13.157 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.063) 0:00:08.179 ********* 2021-03-09T08:50:13.185 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:13.212 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.054) 0:00:08.234 ********* 2021-03-09T08:50:13.231 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-03-09T08:50:13.232 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:13.257 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.044) 0:00:08.279 ********* 2021-03-09T08:50:13.296 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:13.326 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.069) 0:00:08.348 ********* 2021-03-09T08:50:13.441 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-03-09T08:50:13.442 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:50:13.470 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.143) 0:00:08.492 ********* 2021-03-09T08:50:13.515 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:50:13.543 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:13 +0000 (0:00:00.072) 0:00:08.565 ********* 2021-03-09T08:50:13.718 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-09T08:50:13.720 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-09T08:50:13.793 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-09T08:50:13.857 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-09T08:50:13.943 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-09T08:50:14.016 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-09T08:50:14.081 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-09T08:50:14.166 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-09T08:50:14.220 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-09T08:50:14.315 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-09T08:50:14.370 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-09T08:50:14.476 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-09T08:50:14.519 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-09T08:50:14.626 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-09T08:50:14.669 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-09T08:50:14.766 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-09T08:50:14.830 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-09T08:50:14.913 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-09T08:50:14.966 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-09T08:50:15.051 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-09T08:50:15.115 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-09T08:50:15.200 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-09T08:50:15.253 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-09T08:50:15.349 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-09T08:50:15.403 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-09T08:50:15.498 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-09T08:50:15.500 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:15.552 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-09T08:50:15.637 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-09T08:50:15.690 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-09T08:50:15.789 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-09T08:50:15.833 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-09T08:50:15.940 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-09T08:50:15.973 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-09T08:50:16.079 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-09T08:50:16.122 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-09T08:50:16.227 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-09T08:50:16.260 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-09T08:50:16.376 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-09T08:50:16.408 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-09T08:50:16.512 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-09T08:50:16.545 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-09T08:50:16.661 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-09T08:50:16.693 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-09T08:50:16.799 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-03-09T08:50:16.842 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-03-09T08:50:16.948 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-09T08:50:17.001 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-09T08:50:17.096 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-09T08:50:17.138 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-09T08:50:17.232 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-09T08:50:17.285 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-09T08:50:17.380 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-09T08:50:17.434 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-09T08:50:17.530 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-09T08:50:17.573 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-09T08:50:17.668 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-09T08:50:17.721 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-09T08:50:17.825 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-09T08:50:17.868 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-09T08:50:17.974 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-09T08:50:18.006 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-09T08:50:18.122 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-09T08:50:18.165 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-09T08:50:18.271 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-09T08:50:18.303 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-09T08:50:18.412 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-09T08:50:18.455 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-09T08:50:18.562 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-09T08:50:18.594 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-09T08:50:18.710 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-09T08:50:18.743 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-09T08:50:18.860 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-09T08:50:18.883 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-09T08:50:19.010 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-09T08:50:19.033 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-09T08:50:19.159 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-09T08:50:19.171 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-09T08:50:19.307 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T08:50:19.329 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T08:50:19.455 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-09T08:50:19.467 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-09T08:50:19.593 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-09T08:50:19.604 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-09T08:50:19.751 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-09T08:50:19.763 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-09T08:50:19.889 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-09T08:50:19.900 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-09T08:50:20.039 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-09T08:50:20.051 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-09T08:50:20.187 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-09T08:50:20.188 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-09T08:50:20.325 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-09T08:50:20.338 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-09T08:50:20.467 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T08:50:20.479 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T08:50:20.616 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-09T08:50:20.617 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-09T08:50:20.764 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-03-09T08:50:20.765 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-03-09T08:50:20.911 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-09T08:50:20.912 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-09T08:50:21.058 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-09T08:50:21.059 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-09T08:50:21.195 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-09T08:50:21.195 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-09T08:50:21.342 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-09T08:50:21.343 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-09T08:50:21.479 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-09T08:50:21.491 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-09T08:50:21.628 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-09T08:50:21.639 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-09T08:50:21.765 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-03-09T08:50:21.787 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-03-09T08:50:21.924 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-09T08:50:21.935 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-09T08:50:22.061 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-09T08:50:22.083 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-09T08:50:22.198 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-09T08:50:22.231 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-09T08:50:22.347 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-09T08:50:22.369 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-09T08:50:22.495 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-09T08:50:22.517 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-09T08:50:22.632 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-09T08:50:22.664 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-09T08:50:22.779 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-09T08:50:22.812 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-09T08:50:22.926 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T08:50:22.958 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T08:50:23.063 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-09T08:50:23.096 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-09T08:50:23.201 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-09T08:50:23.243 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-09T08:50:23.348 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-09T08:50:23.391 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-09T08:50:23.485 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-09T08:50:23.538 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-09T08:50:23.623 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-09T08:50:23.692 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-09T08:50:23.777 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-09T08:50:23.831 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-09T08:50:23.939 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-09T08:50:24.012 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-09T08:50:24.098 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-09T08:50:24.172 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-09T08:50:24.247 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-09T08:50:24.320 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-09T08:50:24.384 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-09T08:50:24.468 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-09T08:50:24.532 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-09T08:50:24.617 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-09T08:50:24.669 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-09T08:50:24.764 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-09T08:50:24.817 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-09T08:50:24.901 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-09T08:50:24.902 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:24.966 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-09T08:50:25.050 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-09T08:50:25.103 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-09T08:50:25.197 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-09T08:50:25.240 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-09T08:50:25.345 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-09T08:50:25.388 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-09T08:50:25.482 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-09T08:50:25.525 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-09T08:50:25.629 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-09T08:50:25.672 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-09T08:50:25.777 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-09T08:50:25.819 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-09T08:50:25.925 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-09T08:50:25.957 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-09T08:50:26.073 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-09T08:50:26.105 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-09T08:50:26.210 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-09T08:50:26.243 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-09T08:50:26.359 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-09T08:50:26.381 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-09T08:50:26.519 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-09T08:50:26.521 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-09T08:50:26.667 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-09T08:50:26.668 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-09T08:50:26.804 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-09T08:50:26.805 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-09T08:50:26.941 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-09T08:50:26.953 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-09T08:50:27.088 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-09T08:50:27.100 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-09T08:50:27.237 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-09T08:50:27.238 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-09T08:50:27.374 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-09T08:50:27.396 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-09T08:50:27.521 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-09T08:50:27.543 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-09T08:50:27.668 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-09T08:50:27.690 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-09T08:50:27.816 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-09T08:50:27.827 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-09T08:50:27.954 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-09T08:50:27.978 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-09T08:50:28.104 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-09T08:50:28.126 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-09T08:50:28.241 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-09T08:50:28.264 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-09T08:50:28.390 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T08:50:28.413 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-09T08:50:28.529 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-09T08:50:28.551 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T08:50:28.676 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-09T08:50:28.709 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-09T08:50:28.813 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-09T08:50:28.855 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-09T08:50:28.962 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-09T08:50:29.096 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-09T08:50:29.112 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-09T08:50:29.176 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-09T08:50:29.310 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-09T08:50:29.375 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-09T08:50:29.462 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-09T08:50:29.577 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-09T08:50:29.598 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-09T08:50:29.734 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-09T08:50:29.756 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-09T08:50:29.881 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-09T08:50:29.903 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-09T08:50:30.039 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-09T08:50:30.050 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-09T08:50:30.186 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-09T08:50:30.188 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:30.198 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-09T08:50:30.344 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-09T08:50:30.355 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-09T08:50:30.506 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-09T08:50:30.518 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-09T08:50:30.666 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-09T08:50:30.678 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-09T08:50:30.825 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-09T08:50:30.837 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-09T08:50:30.973 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-09T08:50:30.985 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-09T08:50:31.111 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-09T08:50:31.132 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-09T08:50:31.257 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-09T08:50:31.268 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-09T08:50:31.394 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-09T08:50:31.415 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-09T08:50:31.562 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-09T08:50:31.687 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-09T08:50:31.709 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-09T08:50:31.824 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-09T08:50:31.981 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-09T08:50:32.002 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-09T08:50:32.129 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-09T08:50:32.151 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-09T08:50:32.268 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-09T08:50:32.291 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-09T08:50:32.417 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-09T08:50:32.440 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-09T08:50:32.561 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-09T08:50:32.583 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-09T08:50:32.709 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-09T08:50:32.731 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-09T08:50:32.847 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-09T08:50:32.869 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-09T08:50:32.996 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-09T08:50:33.018 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-09T08:50:33.133 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-09T08:50:33.166 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-09T08:50:33.281 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-09T08:50:33.303 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-09T08:50:33.419 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-09T08:50:33.452 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-09T08:50:33.570 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-03-09T08:50:33.592 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-09T08:50:33.716 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-09T08:50:33.738 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-09T08:50:33.867 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-03-09T08:50:33.879 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-03-09T08:50:34.026 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-03-09T08:50:34.027 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-09T08:50:34.249 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-03-09T08:50:34.272 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:34 +0000 (0:00:20.729) 0:00:29.294 ********* 2021-03-09T08:50:34.335 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-03-09T08:50:34.358 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-03-09T08:50:34.386 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-09T08:50:34.473 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:34 +0000 (0:00:00.201) 0:00:29.495 ********* 2021-03-09T08:50:34.496 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:34 +0000 (0:00:00.022) 0:00:29.518 ********* 2021-03-09T08:50:34.552 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-03-09T08:50:34.577 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:34 +0000 (0:00:00.080) 0:00:29.598 ********* 2021-03-09T08:50:34.598 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-03-09T08:50:34.602 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-09T08:50:34.603 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-09T08:50:34.605 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-09T08:50:34.607 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-09T08:50:34.608 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-09T08:50:34.610 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-09T08:50:34.611 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-09T08:50:34.613 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-09T08:50:34.615 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-09T08:50:34.616 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-09T08:50:34.618 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-09T08:50:34.619 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-09T08:50:34.621 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174f 2021-03-09T08:50:34.621 INFO:teuthology.task.ansible.out:e16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-09T08:50:34.653 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:34 +0000 (0:00:00.076) 0:00:29.674 ********* 2021-03-09T08:50:34.674 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-09T08:50:34.675 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-09T08:50:34.677 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-09T08:50:34.679 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\ 2021-03-09T08:50:34.679 INFO:teuthology.task.ansible.out:nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-09T08:50:34.681 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-09T08:50:34.682 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-09T08:50:34.684 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-09T08:50:34.685 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-09T08:50:34.687 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshi 2021-03-09T08:50:34.687 INFO:teuthology.task.ansible.out:t@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-09T08:50:34.689 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-09T08:50:34.690 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-09T08:50:34.692 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-09T08:50:34.694 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, 2021-03-09T08:50:34.695 INFO:teuthology.task.ansible.out: {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-09T08:50:34.725 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:34 +0000 (0:00:00.071) 0:00:29.746 ********* 2021-03-09T08:50:35.898 INFO:teuthology.task.ansible.out:FAILED - RETRYING: Clone the keys repo (5 retries left). 2021-03-09T08:50:35.941 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* ok: [smithi005.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 1, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-09T08:50:35.981 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:35 +0000 (0:00:01.256) 0:00:31.002 ********* 2021-03-09T08:50:36.333 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:36.334 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-09T08:50:36.469 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-09T08:50:36.595 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-09T08:50:36.731 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-03-09T08:50:36.733 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-03-09T08:50:36.866 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-09T08:50:37.003 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-03-09T08:50:37.005 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-03-09T08:50:37.129 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-03-09T08:50:37.264 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-09T08:50:37.390 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-09T08:50:37.401 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:37.537 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-09T08:50:37.539 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-09T08:50:37.673 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-09T08:50:37.674 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-03-09T08:50:37.684 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:37.686 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:37.821 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-09T08:50:37.958 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-09T08:50:37.960 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-09T08:50:38.086 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-09T08:50:38.222 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-09T08:50:38.224 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-03-09T08:50:38.359 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-09T08:50:38.495 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-09T08:50:38.621 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-09T08:50:38.761 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-09T08:50:38.763 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-09T08:50:38.887 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-09T08:50:38.889 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-03-09T08:50:39.054 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-09T08:50:39.181 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-09T08:50:39.318 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-09T08:50:39.321 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-09T08:50:39.455 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-09T08:50:39.590 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-09T08:50:39.727 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-09T08:50:39.864 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-09T08:50:39.991 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-09T08:50:40.128 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-09T08:50:40.130 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-03-09T08:50:40.275 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-09T08:50:40.400 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-09T08:50:40.536 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-09T08:50:40.678 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-09T08:50:40.679 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:40.813 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-09T08:50:40.938 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-09T08:50:41.084 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T08:50:41.085 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:41.219 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-09T08:50:41.355 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-09T08:50:41.491 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-09T08:50:41.493 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-09T08:50:41.616 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-09T08:50:41.618 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-09T08:50:41.752 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-09T08:50:41.897 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-09T08:50:42.033 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-09T08:50:42.170 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-09T08:50:42.172 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T08:50:42.307 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-09T08:50:42.309 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-09T08:50:42.311 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-09T08:50:42.454 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-09T08:50:42.592 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-09T08:50:42.730 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-09T08:50:42.867 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-09T08:50:43.004 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-09T08:50:43.141 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-09T08:50:43.143 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-09T08:50:43.278 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-09T08:50:43.404 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-09T08:50:43.550 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-09T08:50:43.552 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-09T08:50:43.686 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-09T08:50:43.834 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-09T08:50:43.981 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-09T08:50:43.983 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-03-09T08:50:44.117 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-09T08:50:44.252 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-09T08:50:44.389 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-09T08:50:44.517 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T08:50:44.663 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-09T08:50:44.799 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-03-09T08:50:44.945 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-03-09T08:50:45.082 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-03-09T08:50:45.229 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-03-09T08:50:45.365 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-09T08:50:45.503 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-09T08:50:45.639 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-09T08:50:45.785 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-09T08:50:45.923 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-09T08:50:46.071 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-09T08:50:46.220 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-09T08:50:46.347 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-09T08:50:46.484 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-09T08:50:46.623 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-09T08:50:46.760 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-09T08:50:46.907 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-09T08:50:46.910 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-09T08:50:46.929 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-03-09T08:50:46.931 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 2, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-09T08:50:46.968 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:46 +0000 (0:00:10.987) 0:00:41.990 ********* 2021-03-09T08:50:47.049 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:47.051 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-09T08:50:47.180 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-09T08:50:47.295 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-09T08:50:47.316 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-09T08:50:47.432 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-09T08:50:47.454 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-09T08:50:47.568 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-09T08:50:47.591 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-03-09T08:50:47.593 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-03-09T08:50:47.706 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-03-09T08:50:47.708 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-03-09T08:50:47.718 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-09T08:50:47.719 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-03-09T08:50:47.833 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-09T08:50:47.871 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-09T08:50:47.976 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-03-09T08:50:47.978 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-03-09T08:50:47.998 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-09T08:50:48.113 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-03-09T08:50:48.135 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-09T08:50:48.249 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-09T08:50:48.261 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-03-09T08:50:48.377 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-09T08:50:48.388 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:48.400 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-09T08:50:48.515 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-09T08:50:48.517 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-09T08:50:48.537 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-09T08:50:48.651 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-09T08:50:48.653 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-03-09T08:50:48.662 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:48.673 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:48.674 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-09T08:50:48.810 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-09T08:50:48.811 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-09T08:50:48.813 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-09T08:50:48.935 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-09T08:50:48.937 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-09T08:50:48.947 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-09T08:50:49.072 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-09T08:50:49.073 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-09T08:50:49.209 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-09T08:50:49.211 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-09T08:50:49.335 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-09T08:50:49.337 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-09T08:50:49.472 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-09T08:50:49.473 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-09T08:50:49.608 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-09T08:50:49.610 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-09T08:50:49.744 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-09T08:50:49.745 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVK 2021-03-09T08:50:49.746 INFO:teuthology.task.ansible.out:L2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-09T08:50:49.870 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-09T08:50:49.871 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-03-09T08:50:49.881 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-09T08:50:50.007 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-09T08:50:50.009 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-09T08:50:50.145 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-09T08:50:50.147 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-09T08:50:50.282 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-09T08:50:50.284 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-09T08:50:50.285 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-09T08:50:50.422 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-09T08:50:50.558 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-09T08:50:50.561 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-09T08:50:50.687 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-09T08:50:50.689 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-03-09T08:50:50.824 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-09T08:50:50.826 INFO:teuthology.task.ansible.out: ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-09T08:50:50.953 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-09T08:50:50.954 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-03-09T08:50:51.089 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-09T08:50:51.091 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-09T08:50:51.226 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-09T08:50:51.227 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-09T08:50:51.353 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-09T08:50:51.355 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-09T08:50:51.480 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-09T08:50:51.501 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-09T08:50:51.628 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-09T08:50:51.630 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-09T08:50:51.778 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-09T08:50:51.780 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-09T08:50:51.925 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-09T08:50:51.926 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-09T08:50:52.069 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-09T08:50:52.082 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T08:50:52.217 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-09T08:50:52.228 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-09T08:50:52.355 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-03-09T08:50:52.357 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-09T08:50:52.366 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-09T08:50:52.492 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-09T08:50:52.503 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-09T08:50:52.505 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-09T08:50:52.620 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-09T08:50:52.632 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-09T08:50:52.633 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-09T08:50:52.758 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-09T08:50:52.790 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-09T08:50:52.906 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-09T08:50:52.929 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-09T08:50:53.034 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-09T08:50:53.066 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-09T08:50:53.173 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-03-09T08:50:53.206 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-09T08:50:53.208 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T08:50:53.303 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-09T08:50:53.336 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-09T08:50:53.338 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-09T08:50:53.339 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-09T08:50:53.443 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-09T08:50:53.475 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-09T08:50:53.570 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-09T08:50:53.602 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-09T08:50:53.719 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-09T08:50:53.733 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-09T08:50:53.861 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-09T08:50:53.873 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-09T08:50:53.999 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-09T08:50:54.001 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-09T08:50:54.137 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-09T08:50:54.139 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-09T08:50:54.140 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-09T08:50:54.264 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-09T08:50:54.266 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-09T08:50:54.391 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-09T08:50:54.402 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-09T08:50:54.529 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-09T08:50:54.541 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-09T08:50:54.543 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-09T08:50:54.668 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_options" 2021-03-09T08:50:54.670 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-09T08:50:54.679 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-09T08:50:54.805 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-09T08:50:54.876 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-09T08:50:54.901 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:54 +0000 (0:00:07.933) 0:00:49.923 ********* 2021-03-09T08:50:54.963 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-09T08:50:54.965 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-03-09T08:50:54.985 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:50:54.986 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.996 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:54.998 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.009 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.020 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.022 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.032 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.033 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.044 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.097 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:55.099 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-09T08:50:55.192 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:50:55.194 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e 2021-03-09T08:50:55.196 INFO:teuthology.task.ansible.out:1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.206 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.217 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:55.219 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-09T08:50:55.344 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:50:55.346 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-03-09T08:50:55.355 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:55.357 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-09T08:50:55.472 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:50:55.474 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-03-09T08:50:55.483 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.485 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d2 2021-03-09T08:50:55.486 INFO:teuthology.task.ansible.out:8bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.496 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.497 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.507 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.508 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.520 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.521 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.533 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.534 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.544 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.556 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.557 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.568 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.579 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.581 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.590 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.592 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.602 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.604 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.613 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.614 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.625 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.627 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.636 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.638 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-09T08:50:55.649 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.660 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.662 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.672 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.673 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.683 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.694 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.696 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.706 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.708 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.718 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.719 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.730 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.731 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.742 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.744 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.753 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.764 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.766 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-03-09T08:50:55.776 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:50:55.778 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.788 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.790 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.800 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.801 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.812 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.824 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.825 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.835 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.837 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.847 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.847 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.858 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.870 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.881 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.882 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.893 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.894 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.904 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.906 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.919 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.930 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.931 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:55.942 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.944 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.954 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.955 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.967 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.980 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.990 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:55.991 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.001 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.012 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.014 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.024 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.025 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.035 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.037 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.046 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:56.048 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.058 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.060 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.070 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.071 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.081 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.083 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.093 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.104 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.106 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.115 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.117 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.127 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.128 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.138 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.149 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.150 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.160 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.162 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.172 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.174 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.185 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.186 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.196 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.207 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.209 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.218 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.220 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.230 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.241 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.242 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.252 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.254 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.264 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.266 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.278 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.287 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.289 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.299 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.316 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:56.318 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-09T08:50:56.339 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:56 +0000 (0:00:01.437) 0:00:51.361 ********* 2021-03-09T08:50:56.408 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-03-09T08:50:56.410 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:50:56.432 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:50:56 +0000 (0:00:00.092) 0:00:51.454 ********* 2021-03-09T08:50:56.449 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:56.451 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-09T08:50:56.575 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-09T08:50:56.650 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:56.652 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-03-09T08:50:56.707 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:56.709 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-09T08:50:56.791 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi005.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-09T08:50:56.833 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:56.835 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-09T08:50:56.921 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi005.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-09T08:50:56.994 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:56.995 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-09T08:50:57.059 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:57.061 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-09T08:50:57.147 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-09T08:50:57.212 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi005.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-09T08:50:57.288 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:57.290 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-09T08:50:57.352 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:57.355 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-09T08:50:57.487 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:57.489 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-09T08:50:57.513 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:57.515 INFO:teuthology.task.ansible.out:TASK [Remove revoked users] **************************************************** ok: [smithi005.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-09T08:50:57.634 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:57.636 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-09T08:50:57.677 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:57.679 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-09T08:50:57.784 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:57.786 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-09T08:50:57.817 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:57.819 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-09T08:50:57.949 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:57.951 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-09T08:50:57.953 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-09T08:50:57.982 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:57.984 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-09T08:50:58.102 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:58.104 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-09T08:50:58.124 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:58.126 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-09T08:50:58.243 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:58.244 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-09T08:50:58.275 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:58.278 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-09T08:50:58.390 INFO:teuthology.task.ansible.out: 2021-03-09T08:50:58.391 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:58.393 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-09T08:50:58.427 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:58.428 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-09T08:50:58.564 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:58.566 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-09T08:50:58.576 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:58.578 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-09T08:50:58.707 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:58.709 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-03-09T08:50:58.713 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-03-09T08:50:58.720 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:58.723 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-09T08:50:58.852 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:58.854 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-09T08:50:58.856 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} TASK [Remove revoked users] **************************************************** ok: [smithi005.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-09T08:50:58.991 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:58.993 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-09T08:50:59.005 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi005.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-09T08:50:59.131 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:59.133 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-09T08:50:59.153 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:59.155 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-09T08:50:59.259 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:59.261 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-09T08:50:59.291 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:59.293 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-09T08:50:59.397 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:59.399 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-03-09T08:50:59.419 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:59.421 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-09T08:50:59.537 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:59.539 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-09T08:50:59.561 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:59.562 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-09T08:50:59.669 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:59.671 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-09T08:50:59.702 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:59.704 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-09T08:50:59.811 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:59.813 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-09T08:50:59.844 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:59.846 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-09T08:50:59.961 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:50:59.963 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-09T08:50:59.965 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-09T08:50:59.994 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:50:59.996 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-09T08:51:00.100 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:00.102 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-09T08:51:00.133 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:00.135 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-09T08:51:00.241 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:00.243 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-09T08:51:00.274 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:00.276 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-09T08:51:00.380 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:00.382 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-09T08:51:00.413 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:00.415 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-09T08:51:00.508 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:00.510 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-09T08:51:00.551 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:00.553 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-09T08:51:00.636 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:00.638 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-09T08:51:00.690 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:00.692 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-09T08:51:00.776 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:00.778 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-09T08:51:00.830 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:00.832 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-09T08:51:00.914 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:00.916 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-09T08:51:00.918 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T08:51:00.979 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:00.981 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-09T08:51:01.044 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:01.046 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-09T08:51:01.129 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:01.132 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-09T08:51:01.184 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:01.185 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-09T08:51:01.269 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:01.271 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-09T08:51:01.312 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:01.314 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-09T08:51:01.408 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:01.409 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-09T08:51:01.451 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:01.453 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-09T08:51:01.536 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:01.539 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-09T08:51:01.597 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:01.599 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-03-09T08:51:01.692 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:01.693 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-09T08:51:01.724 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:01.726 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-09T08:51:01.829 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:01.830 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-09T08:51:01.873 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:01.874 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-03-09T08:51:01.970 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:01.972 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-03-09T08:51:02.014 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:02.016 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-09T08:51:02.111 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:02.114 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-03-09T08:51:02.155 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:02.157 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-03-09T08:51:02.253 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi005.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-09T08:51:02.287 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:02.290 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-09T08:51:02.395 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:02.397 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-09T08:51:02.418 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:02.420 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-09T08:51:02.537 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:02.539 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-09T08:51:02.571 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:02.572 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-09T08:51:02.680 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:02.682 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-09T08:51:02.713 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:02.715 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-09T08:51:02.824 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:02.826 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-09T08:51:02.846 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:02.848 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-09T08:51:02.964 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:02.966 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-09T08:51:02.986 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:02.988 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-09T08:51:03.114 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:03.116 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-09T08:51:03.127 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:03.128 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-09T08:51:03.246 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:03.247 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-09T08:51:03.269 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:03.271 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-09T08:51:03.397 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:03.400 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-09T08:51:03.409 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:03.412 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-09T08:51:03.538 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:03.540 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-09T08:51:03.550 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:03.551 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-03-09T08:51:03.553 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-09T08:51:03.562 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:03.604 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:03 +0000 (0:00:07.172) 0:00:58.626 ********* 2021-03-09T08:51:03.696 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:03.698 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-09T08:51:03.833 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-09T08:51:03.971 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-09T08:51:04.078 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-03-09T08:51:04.080 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279863.7208366-12647-242063611496358/source", "state": "file", "uid": 0} 2021-03-09T08:51:04.111 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:04 +0000 (0:00:00.507) 0:00:59.133 ********* 2021-03-09T08:51:04.131 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:04.133 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-09T08:51:04.185 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi005.front.sepia.ceph.com] => { "msg": "Host smithi005.front.sepia.ceph.com is running Ubuntu 20.04 (focal)" } 2021-03-09T08:51:04.211 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:04 +0000 (0:00:00.099) 0:00:59.233 ********* 2021-03-09T08:51:04.270 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:04.273 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-09T08:51:04.292 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi005.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-03-09T08:51:04.318 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:04 +0000 (0:00:00.106) 0:00:59.339 ********* 2021-03-09T08:51:04.387 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-03-09T08:51:04.389 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:04.410 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:04 +0000 (0:00:00.092) 0:00:59.432 ********* 2021-03-09T08:51:04.418 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:04.420 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-03-09T08:51:04.482 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:04.510 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:04 +0000 (0:00:00.099) 0:00:59.532 ********* 2021-03-09T08:51:04.561 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:04.563 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-09T08:51:04.583 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:04.613 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:04 +0000 (0:00:00.103) 0:00:59.635 ********* 2021-03-09T08:51:04.695 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:04.697 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-09T08:51:04.707 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-03-09T08:51:04.709 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:04.732 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:04 +0000 (0:00:00.118) 0:00:59.754 ********* 2021-03-09T08:51:04.812 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-03-09T08:51:04.814 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:04.840 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:04 +0000 (0:00:00.108) 0:00:59.862 ********* 2021-03-09T08:51:04.851 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:04.852 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-09T08:51:04.916 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-03-09T08:51:04.918 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:04.939 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:04 +0000 (0:00:00.099) 0:00:59.961 ********* 2021-03-09T08:51:04.979 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:04.980 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-09T08:51:05.004 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:05.027 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:05 +0000 (0:00:00.087) 0:01:00.049 ********* 2021-03-09T08:51:05.096 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-03-09T08:51:05.098 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:05.121 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:05 +0000 (0:00:00.093) 0:01:00.142 ********* 2021-03-09T08:51:05.128 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:05.130 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-09T08:51:05.204 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-03-09T08:51:05.206 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi005.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:51:05.228 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:05 +0000 (0:00:00.107) 0:01:00.250 ********* 2021-03-09T08:51:05.256 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:05.258 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-09T08:51:05.300 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:05.327 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:05 +0000 (0:00:00.098) 0:01:00.349 ********* 2021-03-09T08:51:05.387 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:05.389 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-09T08:51:05.409 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:05.437 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:05 +0000 (0:00:00.110) 0:01:00.459 ********* 2021-03-09T08:51:05.517 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-03-09T08:51:05.519 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:51:05.541 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:05 +0000 (0:00:00.104) 0:01:00.563 ********* 2021-03-09T08:51:05.549 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:05.551 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-09T08:51:05.616 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:05.641 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:05 +0000 (0:00:00.099) 0:01:00.663 ********* 2021-03-09T08:51:05.669 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:05.671 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-09T08:51:05.723 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi005.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:51:05.752 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:05 +0000 (0:00:00.111) 0:01:00.774 ********* 2021-03-09T08:51:05.804 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:05.806 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-09T08:51:05.837 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-03-09T08:51:05.838 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:05.860 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:05 +0000 (0:00:00.107) 0:01:00.882 ********* 2021-03-09T08:51:05.930 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:05.932 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi146.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6Qx 2021-03-09T08:51:05.934 INFO:teuthology.task.ansible.out:C6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} TASK [common : List CDN releases available to system] ************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:05.958 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:05 +0000 (0:00:00.098) 0:01:00.980 ********* 2021-03-09T08:51:06.041 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-03-09T08:51:06.044 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.076 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:06 +0000 (0:00:00.117) 0:01:01.097 ********* 2021-03-09T08:51:06.084 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:51:06.086 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-09T08:51:06.173 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:06 +0000 (0:00:00.097) 0:01:01.195 ********* 2021-03-09T08:51:06.180 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-09T08:51:06.182 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.216 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:06 +0000 (0:00:00.042) 0:01:01.238 ********* 2021-03-09T08:51:06.268 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:51:06.270 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.280 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.281 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.292 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.293 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi005.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:51:06.293 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:51:06.294 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.321 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:06 +0000 (0:00:00.104) 0:01:01.343 ********* 2021-03-09T08:51:06.334 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.336 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.337 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.339 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.414 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.438 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:06 +0000 (0:00:00.117) 0:01:01.460 ********* 2021-03-09T08:51:06.466 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:51:06.468 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-03-09T08:51:06.470 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.481 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.513 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.537 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:06 +0000 (0:00:00.099) 0:01:01.559 ********* 2021-03-09T08:51:06.608 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-03-09T08:51:06.609 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.632 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:06 +0000 (0:00:00.094) 0:01:01.654 ********* 2021-03-09T08:51:06.640 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:51:06.641 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-03-09T08:51:06.704 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-03-09T08:51:06.706 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.729 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:06 +0000 (0:00:00.096) 0:01:01.751 ********* 2021-03-09T08:51:06.747 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:51:06.749 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-03-09T08:51:06.761 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.774 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.776 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.785 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.788 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.797 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.799 INFO:teuthology.task.ansible.out: skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.809 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.811 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:51:06.813 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.835 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:06 +0000 (0:00:00.106) 0:01:01.857 ********* 2021-03-09T08:51:06.837 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:06.844 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.846 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.846 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.848 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.858 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.860 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.870 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.871 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.882 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.893 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.895 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.907 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.908 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.910 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:51:06.910 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.939 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:06 +0000 (0:00:00.103) 0:01:01.961 ********* 2021-03-09T08:51:06.947 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.947 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.949 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.951 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.962 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.963 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.974 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.975 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.986 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.987 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:06.998 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.010 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.011 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-03-09T08:51:07.013 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:51:07.015 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.039 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:07 +0000 (0:00:00.099) 0:01:02.061 ********* 2021-03-09T08:51:07.046 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.048 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.058 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.060 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.070 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.071 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:07.082 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.083 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.084 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.095 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.097 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.107 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-03-09T08:51:07.109 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.133 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:07 +0000 (0:00:00.094) 0:01:02.155 ********* 2021-03-09T08:51:07.141 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.142 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.144 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.144 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.145 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.156 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.167 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.169 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.179 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.180 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.190 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.191 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.203 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.204 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.228 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:07 +0000 (0:00:00.095) 0:01:02.250 ********* 2021-03-09T08:51:07.236 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:51:07.237 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.238 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.239 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.241 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.252 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.254 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.264 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.266 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.277 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.288 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.290 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.300 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.302 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.312 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.323 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.325 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.348 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:07 +0000 (0:00:00.119) 0:01:02.370 ********* 2021-03-09T08:51:07.356 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.357 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.358 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.359 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.370 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.372 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.382 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.383 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.393 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.394 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.405 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.416 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.418 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.428 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.429 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.440 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.442 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:51:07.442 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.443 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.466 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:07 +0000 (0:00:00.117) 0:01:02.488 ********* 2021-03-09T08:51:07.473 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.475 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.476 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.488 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.489 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.499 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.511 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.523 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.525 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.535 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.546 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.547 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.557 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.559 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* skipping: [smithi005.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.559 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.583 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:07 +0000 (0:00:00.116) 0:01:02.605 ********* 2021-03-09T08:51:07.590 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.591 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.593 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.594 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.594 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.609 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.651 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:07 +0000 (0:00:00.068) 0:01:02.673 ********* 2021-03-09T08:51:07.658 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T08:51:07.660 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.686 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:07 +0000 (0:00:00.034) 0:01:02.708 ********* 2021-03-09T08:51:07.746 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-03-09T08:51:07.748 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:07.772 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:07 +0000 (0:00:00.086) 0:01:02.794 ********* 2021-03-09T08:51:07.916 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:07.918 INFO:teuthology.task.ansible.out:TASK [common : Get the current timezone] *************************************** ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.011606", "end": "2021-03-09 08:51:07.591891", "rc": 0, "start": "2021-03-09 08:51:07.580285", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-09T08:51:07.945 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:07 +0000 (0:00:00.172) 0:01:02.967 ********* 2021-03-09T08:51:08.004 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:08.006 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-03-09T08:51:08.026 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:08.049 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:08 +0000 (0:00:00.104) 0:01:03.071 ********* 2021-03-09T08:51:08.130 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:08.132 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-09T08:51:08.142 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-03-09T08:51:08.144 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:08.165 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:08 +0000 (0:00:00.116) 0:01:03.187 ********* 2021-03-09T08:51:08.269 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:08.271 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-09T08:51:08.407 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-09T08:51:08.544 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-09T08:51:08.681 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-09T08:51:08.818 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-09T08:51:08.944 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-09T08:51:08.987 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-09T08:51:08.989 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "name": "sleep.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "sleep.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "sleep.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit sleep.target is masked.\"", "LoadState": "masked", "Names": "sleep.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-03-09T08:51:09.083 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi146.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-09T08:51:09.167 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-09T08:51:09.169 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "name": "suspend.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "suspend.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "suspend.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit suspend.target is masked.\"", "LoadState": "masked", "Names": "suspend.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-03-09T08:51:09.220 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:09.222 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-09T08:51:09.358 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-09T08:51:09.360 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-09T08:51:09.362 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "name": "hibernate.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "hibernate.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "hibernate.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit hibernate.target is masked.\"", "LoadState": "masked", "Names": "hibernate.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-03-09T08:51:09.488 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:09.490 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-09T08:51:09.542 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-09T08:51:09.544 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "name": "hybrid-sleep.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "hybrid-sleep.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "hybrid-sleep.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit hybrid-sleep.target is masked.\"", "LoadState": "masked", "Names": "hybrid-sleep.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-03-09T08:51:09.572 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:09 +0000 (0:00:01.406) 0:01:04.594 ********* 2021-03-09T08:51:09.623 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:09.625 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-09T08:51:09.647 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-09T08:51:09.649 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:09.672 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:09 +0000 (0:00:00.099) 0:01:04.694 ********* 2021-03-09T08:51:09.741 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-03-09T08:51:09.743 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:09.766 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:09 +0000 (0:00:00.094) 0:01:04.788 ********* 2021-03-09T08:51:09.773 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:09.775 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-09T08:51:09.852 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:09.876 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:09 +0000 (0:00:00.109) 0:01:04.898 ********* 2021-03-09T08:51:09.896 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:09.898 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-09T08:51:09.951 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-03-09T08:51:09.953 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:09.994 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:09 +0000 (0:00:00.118) 0:01:05.016 ********* 2021-03-09T08:51:10.028 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:10.030 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-09T08:51:10.092 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:10.116 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:10 +0000 (0:00:00.122) 0:01:05.138 ********* 2021-03-09T08:51:10.177 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:10.179 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-09T08:51:10.311 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-09T08:51:10.448 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-09T08:51:10.575 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-09T08:51:10.721 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-09T08:51:10.838 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-03-09T08:51:10.840 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.571376", "end": "2021-03-09 08:51:10.508067", "rc": 0, "start": "2021-03-09 08:51:09.936691", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:10.878 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:10 +0000 (0:00:00.761) 0:01:05.899 ********* 2021-03-09T08:51:10.887 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:10.889 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-09T08:51:10.983 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-09T08:51:11.025 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:11.050 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:11 +0000 (0:00:00.171) 0:01:06.071 ********* 2021-03-09T08:51:11.119 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:11.120 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-09T08:51:11.256 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-09T08:51:11.267 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-03-09T08:51:11.269 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.019831", "end": "2021-03-09 08:51:10.934797", "rc": 0, "start": "2021-03-09 08:51:10.914966", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:11.291 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:11 +0000 (0:00:00.242) 0:01:06.313 ********* 2021-03-09T08:51:11.393 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:11.395 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-09T08:51:11.531 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-09T08:51:11.667 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-09T08:51:11.803 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-09T08:51:11.942 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-09T08:51:12.077 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-09T08:51:12.204 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-09T08:51:12.341 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-09T08:51:12.477 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-09T08:51:12.613 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-09T08:51:12.752 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-09T08:51:12.883 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-09T08:51:13.021 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-03-09T08:51:13.147 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-03-09T08:51:13.285 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-09T08:51:13.413 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-09T08:51:13.541 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-09T08:51:13.689 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-09T08:51:13.828 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-09T08:51:13.860 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-03-09T08:51:13.862 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1615279787, "cache_updated": false, "changed": false} 2021-03-09T08:51:13.887 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:13 +0000 (0:00:02.595) 0:01:08.909 ********* 2021-03-09T08:51:13.970 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:51:13.973 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-09T08:51:14.099 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-09T08:51:14.237 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-09T08:51:14.374 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-09T08:51:14.514 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-09T08:51:14.557 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:14 +0000 (0:00:00.669) 0:01:09.579 ********* 2021-03-09T08:51:14.606 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-09T08:51:14.608 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"cache_update_time": 1615279787, "cache_updated": false, "changed": false} 2021-03-09T08:51:14.643 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:14 +0000 (0:00:00.086) 0:01:09.665 ********* 2021-03-09T08:51:14.716 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-03-09T08:51:14.717 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:14.739 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:14 +0000 (0:00:00.095) 0:01:09.761 ********* 2021-03-09T08:51:14.974 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-03-09T08:51:14.976 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279874.6426601-13421-158826125965879/source", "state": "file", "uid": 0} 2021-03-09T08:51:15.001 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:14 +0000 (0:00:00.262) 0:01:10.023 ********* 2021-03-09T08:51:15.070 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-03-09T08:51:15.072 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => { "msg": "Host smithi146.front.sepia.ceph.com is running Ubuntu 20.04 (focal)" } 2021-03-09T08:51:15.094 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:15 +0000 (0:00:00.093) 0:01:10.116 ********* 2021-03-09T08:51:15.185 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-03-09T08:51:15.187 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-03-09T08:51:15.211 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:15 +0000 (0:00:00.116) 0:01:10.233 ********* 2021-03-09T08:51:15.280 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:15.305 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:15 +0000 (0:00:00.093) 0:01:10.326 ********* 2021-03-09T08:51:15.374 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-03-09T08:51:15.376 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:15.405 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:15 +0000 (0:00:00.100) 0:01:10.427 ********* 2021-03-09T08:51:15.412 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-03-09T08:51:15.413 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-03-09T08:51:15.440 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:15 +0000 (0:00:00.034) 0:01:10.462 ********* 2021-03-09T08:51:15.468 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-03-09T08:51:15.469 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:15.493 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:15 +0000 (0:00:00.052) 0:01:10.514 ********* 2021-03-09T08:51:15.562 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:15.564 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:15.586 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:15 +0000 (0:00:00.092) 0:01:10.607 ********* 2021-03-09T08:51:15.656 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:15.657 INFO:teuthology.task.ansible.out:TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:15.682 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:15 +0000 (0:00:00.096) 0:01:10.704 ********* 2021-03-09T08:51:15.751 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-03-09T08:51:15.753 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:15.775 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:15 +0000 (0:00:00.092) 0:01:10.796 ********* 2021-03-09T08:51:15.855 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-03-09T08:51:15.857 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:15.878 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:15 +0000 (0:00:00.103) 0:01:10.900 ********* 2021-03-09T08:51:15.958 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-03-09T08:51:15.960 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:15.987 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:15 +0000 (0:00:00.108) 0:01:11.008 ********* 2021-03-09T08:51:16.076 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-03-09T08:51:16.078 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi146.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:51:16.101 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.113) 0:01:11.122 ********* 2021-03-09T08:51:16.181 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-03-09T08:51:16.183 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"cache_update_time": 1615279787, "cache_updated": false, "changed": false} TASK [common : Set have_entitlements] ****************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:16.206 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.105) 0:01:11.227 ********* 2021-03-09T08:51:16.229 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.022) 0:01:11.250 ********* 2021-03-09T08:51:16.299 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-03-09T08:51:16.301 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:16.328 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.099) 0:01:11.350 ********* 2021-03-09T08:51:16.390 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi146.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:51:16.415 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.087) 0:01:11.437 ********* 2021-03-09T08:51:16.484 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-03-09T08:51:16.485 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:16.508 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.092) 0:01:11.529 ********* 2021-03-09T08:51:16.576 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-03-09T08:51:16.578 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:51:16.601 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.093) 0:01:11.623 ********* 2021-03-09T08:51:16.618 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T08:51:16.620 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-03-09T08:51:16.673 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-03-09T08:51:16.675 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:16.699 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.097) 0:01:11.721 ********* 2021-03-09T08:51:16.770 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-03-09T08:51:16.772 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:16.794 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.095) 0:01:11.816 ********* 2021-03-09T08:51:16.875 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-03-09T08:51:16.877 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:16.904 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.109) 0:01:11.926 ********* 2021-03-09T08:51:16.956 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T08:51:16.956 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-03-09T08:51:16.979 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-09T08:51:16.981 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:17.004 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:16 +0000 (0:00:00.099) 0:01:12.025 ********* 2021-03-09T08:51:17.073 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi146.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:51:17.100 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.096) 0:01:12.122 ********* 2021-03-09T08:51:17.182 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:17.216 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.115) 0:01:12.238 ********* 2021-03-09T08:51:17.269 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T08:51:17.271 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-03-09T08:51:17.299 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.082) 0:01:12.320 ********* 2021-03-09T08:51:17.306 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-03-09T08:51:17.308 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:17.334 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.035) 0:01:12.356 ********* 2021-03-09T08:51:17.414 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:17.438 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.104) 0:01:12.460 ********* 2021-03-09T08:51:17.444 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-03-09T08:51:17.446 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:51:17.470 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.031) 0:01:12.492 ********* 2021-03-09T08:51:17.498 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-03-09T08:51:17.500 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:17.528 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.058) 0:01:12.550 ********* 2021-03-09T08:51:17.635 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:17.667 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.139) 0:01:12.689 ********* 2021-03-09T08:51:17.766 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-03-09T08:51:17.768 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:17.798 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.130) 0:01:12.820 ********* 2021-03-09T08:51:17.872 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-03-09T08:51:17.874 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:51:17.898 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.099) 0:01:12.919 ********* 2021-03-09T08:51:17.906 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T08:51:17.908 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-03-09T08:51:17.972 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:17.998 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:17 +0000 (0:00:00.100) 0:01:13.020 ********* 2021-03-09T08:51:18.089 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-03-09T08:51:18.091 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:18.114 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.116) 0:01:13.136 ********* 2021-03-09T08:51:18.185 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T08:51:18.187 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-03-09T08:51:18.207 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:18.235 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.120) 0:01:13.256 ********* 2021-03-09T08:51:18.315 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-09T08:51:18.317 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:18.328 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:18.359 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.124) 0:01:13.381 ********* 2021-03-09T08:51:18.442 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-03-09T08:51:18.445 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:18.473 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.114) 0:01:13.495 ********* 2021-03-09T08:51:18.481 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T08:51:18.483 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-03-09T08:51:18.513 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.039) 0:01:13.534 ********* 2021-03-09T08:51:18.543 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T08:51:18.544 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:18.576 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.063) 0:01:13.598 ********* 2021-03-09T08:51:18.595 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-09T08:51:18.597 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-03-09T08:51:18.625 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.048) 0:01:13.647 ********* 2021-03-09T08:51:18.718 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-09T08:51:18.720 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:18.745 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.119) 0:01:13.766 ********* 2021-03-09T08:51:18.780 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-09T08:51:18.782 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.016241", "end": "2021-03-09 08:51:19.175192", "rc": 0, "start": "2021-03-09 08:51:19.158951", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-09T08:51:18.807 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.062) 0:01:13.829 ********* 2021-03-09T08:51:18.827 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-09T08:51:18.829 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:51:18.862 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.054) 0:01:13.883 ********* 2021-03-09T08:51:18.893 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-03-09T08:51:18.895 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:18.923 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.061) 0:01:13.945 ********* 2021-03-09T08:51:18.951 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-09T08:51:18.953 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:18.962 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:18.964 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:18.973 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:18.998 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:18 +0000 (0:00:00.074) 0:01:14.020 ********* 2021-03-09T08:51:19.015 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-03-09T08:51:19.017 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:19.040 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:19 +0000 (0:00:00.042) 0:01:14.062 ********* 2021-03-09T08:51:19.080 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-09T08:51:19.081 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:19.117 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:19 +0000 (0:00:00.076) 0:01:14.138 ********* 2021-03-09T08:51:19.326 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-09T08:51:19.328 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "name": "sleep.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "sleep.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "sleep.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit sleep.target is masked.\"", "LoadState": "masked", "Names": "sleep.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-03-09T08:51:19.521 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "name": "suspend.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "suspend.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "suspend.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit suspend.target is masked.\"", "LoadState": "masked", "Names": "suspend.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-03-09T08:51:19.703 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "name": "hibernate.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "hibernate.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "hibernate.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit hibernate.target is masked.\"", "LoadState": "masked", "Names": "hibernate.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-03-09T08:51:19.860 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-03-09T08:51:19.898 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"cache_update_time": 1615279787, "cache_updated": false, "changed": false} 2021-03-09T08:51:19.915 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:19 +0000 (0:00:00.798) 0:01:14.937 ********* 2021-03-09T08:51:19.935 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-09T08:51:19.938 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "name": "hybrid-sleep.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "hybrid-sleep.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "hybrid-sleep.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit hybrid-sleep.target is masked.\"", "LoadState": "masked", "Names": "hybrid-sleep.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-03-09T08:51:19.987 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:19 +0000 (0:00:00.071) 0:01:15.008 ********* 2021-03-09T08:51:20.098 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-09T08:51:20.100 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:20.124 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:20 +0000 (0:00:00.137) 0:01:15.146 ********* 2021-03-09T08:51:20.215 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-03-09T08:51:20.217 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:20.240 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:20 +0000 (0:00:00.116) 0:01:15.262 ********* 2021-03-09T08:51:20.313 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-03-09T08:51:20.315 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:20.337 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:20 +0000 (0:00:00.096) 0:01:15.358 ********* 2021-03-09T08:51:20.418 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-03-09T08:51:20.420 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:20.443 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:20 +0000 (0:00:00.106) 0:01:15.465 ********* 2021-03-09T08:51:20.451 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-03-09T08:51:20.453 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-03-09T08:51:20.487 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:20 +0000 (0:00:00.043) 0:01:15.508 ********* 2021-03-09T08:51:20.519 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:20.521 INFO:teuthology.task.ansible.out:TASK [common : Refresh repos] ************************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:20.548 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:20 +0000 (0:00:00.061) 0:01:15.570 ********* 2021-03-09T08:51:20.550 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:20.930 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-03-09T08:51:20.932 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-03-09T08:51:20.957 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:20 +0000 (0:00:00.409) 0:01:15.979 ********* 2021-03-09T08:51:21.110 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-03-09T08:51:21.112 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.421315", "end": "2021-03-09 08:51:21.507859", "rc": 0, "start": "2021-03-09 08:51:21.086544", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:21.136 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:21 +0000 (0:00:00.178) 0:01:16.157 ********* 2021-03-09T08:51:21.174 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-09T08:51:21.179 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-03-09T08:51:21.206 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-03-09T08:51:21.208 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:21.231 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:21 +0000 (0:00:00.095) 0:01:16.253 ********* 2021-03-09T08:51:21.293 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-09T08:51:21.294 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-03-09T08:51:21.321 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:21 +0000 (0:00:00.089) 0:01:16.342 ********* 2021-03-09T08:51:21.412 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:21.437 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:21 +0000 (0:00:00.116) 0:01:16.459 ********* 2021-03-09T08:51:21.445 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-03-09T08:51:21.446 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.020416", "end": "2021-03-09 08:51:21.830596", "rc": 0, "start": "2021-03-09 08:51:21.810180", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:21.473 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:21 +0000 (0:00:00.035) 0:01:16.495 ********* 2021-03-09T08:51:21.511 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-03-09T08:51:21.513 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:51:21.537 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:21 +0000 (0:00:00.063) 0:01:16.559 ********* 2021-03-09T08:51:21.627 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-03-09T08:51:21.629 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:21.652 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:21 +0000 (0:00:00.114) 0:01:16.673 ********* 2021-03-09T08:51:22.057 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-03-09T08:51:22.059 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-03-09T08:51:22.088 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:22 +0000 (0:00:00.435) 0:01:17.109 ********* 2021-03-09T08:51:22.351 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-03-09T08:51:22.352 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 08:48:19 UTC", "ActiveEnterTimestampMonotonic": "13546923", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target sysinit.target -.mount remote-fs.target var-run.mount nss-lookup.target time-sync.target system.slice systemd-journald.socket basic.target local-fs.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 08:48:19 UTC", "AssertTimestampMonotonic": "13544651", "Before": "multi-user.target getty@tty1.service xdm.service plymouth-quit.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 08:48:19 UTC", "Co 2021-03-09T08:51:22.355 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "13544651", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "702", "ExecMainStartTimestamp": "Tue 2021-03-09 08:48:19 UTC", "ExecMainStartTimestampMonotonic": "13546709", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPostEx": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nagios/nrpe.pid ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", 2021-03-09T08:51:22.356 INFO:teuthology.task.ansible.out: "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 08:48:19 UTC", "InactiveExitTimestampMonotonic": "13546923", "InvocationID": "d6935aaef85c489da35dfc9260ce9d78", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127805", "LimitNPROCSoft": "127805", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127805", "LimitSIGPENDINGSoft": "127805", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "702", "MemoryAccounting": "yes", "MemoryCurrent": "5697536", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "M 2021-03-09T08:51:22.358 INFO:teuthology.task.ansible.out:emoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nagios/nrpe.pid", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:48:19 UTC", "StateChangeTimestampMonotonic": "13546923", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "S 2021-03-09T08:51:22.360 INFO:teuthology.task.ansible.out:yslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "38341", "TimeoutAbortUSec": "1min", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "110", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-09T08:51:22.382 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:22 +0000 (0:00:00.294) 0:01:17.404 ********* 2021-03-09T08:51:22.462 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:22.486 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:22 +0000 (0:00:00.104) 0:01:17.508 ********* 2021-03-09T08:51:22.565 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:22.567 INFO:teuthology.task.ansible.out:TASK [common : nrpe - Install SELinux dependencies] **************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:22.589 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:22 +0000 (0:00:00.103) 0:01:17.611 ********* 2021-03-09T08:51:22.659 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-03-09T08:51:22.661 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:22.682 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:22 +0000 (0:00:00.093) 0:01:17.704 ********* 2021-03-09T08:51:22.773 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:22.797 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:22 +0000 (0:00:00.114) 0:01:17.819 ********* 2021-03-09T08:51:22.868 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-03-09T08:51:22.870 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:22.905 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:22 +0000 (0:00:00.107) 0:01:17.926 ********* 2021-03-09T08:51:22.987 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-03-09T08:51:22.988 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:23.019 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.114) 0:01:18.041 ********* 2021-03-09T08:51:23.102 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-03-09T08:51:23.104 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:23.129 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.109) 0:01:18.151 ********* 2021-03-09T08:51:23.210 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-03-09T08:51:23.212 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:23.236 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.106) 0:01:18.257 ********* 2021-03-09T08:51:23.318 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-03-09T08:51:23.320 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:23.351 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.115) 0:01:18.373 ********* 2021-03-09T08:51:23.435 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-03-09T08:51:23.437 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:23.460 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.108) 0:01:18.482 ********* 2021-03-09T08:51:23.540 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-03-09T08:51:23.542 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-03-09T08:51:23.564 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.104) 0:01:18.586 ********* 2021-03-09T08:51:23.655 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-03-09T08:51:23.657 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-03-09T08:51:23.684 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.119) 0:01:18.705 ********* 2021-03-09T08:51:23.774 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-03-09T08:51:23.776 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "docker.io", "python3-nose", "python3-virtualenv", "python3-configobj", "python3-gevent", "python3-numpy", "python3-matplotlib", "python3-setuptools"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml"} 2021-03-09T08:51:23.799 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.114) 0:01:18.820 ********* 2021-03-09T08:51:23.831 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-03-09T08:51:23.834 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1615279794, "cache_updated": false, "changed": false} 2021-03-09T08:51:23.860 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.060) 0:01:18.880 ********* 2021-03-09T08:51:23.897 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-09T08:51:23.899 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-09T08:51:23.924 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:23 +0000 (0:00:00.065) 0:01:18.946 ********* 2021-03-09T08:51:24.071 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-03-09T08:51:24.073 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-09T08:51:24.094 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:24 +0000 (0:00:00.170) 0:01:19.116 ********* 2021-03-09T08:51:24.293 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-03-09T08:51:24.295 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-09T08:51:24.317 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:24 +0000 (0:00:00.222) 0:01:19.339 ********* 2021-03-09T08:51:24.397 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-03-09T08:51:24.399 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:24.423 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:24 +0000 (0:00:00.106) 0:01:19.445 ********* 2021-03-09T08:51:24.556 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-09T08:51:24.558 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1615279794, "cache_updated": false, "changed": false} 2021-03-09T08:51:24.590 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:24 +0000 (0:00:00.166) 0:01:19.612 ********* 2021-03-09T08:51:24.653 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-03-09T08:51:24.655 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:24.680 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:24 +0000 (0:00:00.089) 0:01:19.702 ********* 2021-03-09T08:51:24.786 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-03-09T08:51:24.788 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:24.812 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:24 +0000 (0:00:00.131) 0:01:19.833 ********* 2021-03-09T08:51:25.033 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-03-09T08:51:25.035 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-03-09T08:51:25.059 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:25 +0000 (0:00:00.247) 0:01:20.081 ********* 2021-03-09T08:51:25.445 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-03-09T08:51:25.446 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-03-09T08:51:25.470 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:25 +0000 (0:00:00.410) 0:01:20.491 ********* 2021-03-09T08:51:25.756 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-09T08:51:25.757 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-03-09T08:51:25.790 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:25 +0000 (0:00:00.320) 0:01:20.812 ********* 2021-03-09T08:51:25.892 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi005"}, "changed": false} 2021-03-09T08:51:25.920 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:25 +0000 (0:00:00.130) 0:01:20.942 ********* 2021-03-09T08:51:25.922 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:26.152 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-03-09T08:51:26.154 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1615279794, "cache_updated": false, "changed": false} 2021-03-09T08:51:26.178 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:26 +0000 (0:00:00.257) 0:01:21.200 ********* 2021-03-09T08:51:26.787 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-03-09T08:51:26.788 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi005", "ansible_hostname": "smithi005", "ansible_nodename": "smithi005"}, "changed": false, "name": "smithi005"} 2021-03-09T08:51:26.820 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:26 +0000 (0:00:00.641) 0:01:21.841 ********* 2021-03-09T08:51:26.830 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T08:51:26.832 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-03-09T08:51:27.178 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-03-09T08:51:27.372 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-03-09T08:51:27.374 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-03-09T08:51:27.397 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.577) 0:01:22.418 ********* 2021-03-09T08:51:27.488 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T08:51:27.490 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-03-09T08:51:27.517 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.120) 0:01:22.539 ********* 2021-03-09T08:51:27.661 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-03-09T08:51:27.663 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:51:27.787 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.270) 0:01:22.809 ********* 2021-03-09T08:51:27.941 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-03-09T08:51:27.943 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-03-09T08:51:27.975 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:27 +0000 (0:00:00.187) 0:01:22.997 ********* 2021-03-09T08:51:28.165 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi146.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-03-09T08:51:28.419 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-03-09T08:51:28.421 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-09T08:51:28.443 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:28 +0000 (0:00:00.467) 0:01:23.465 ********* 2021-03-09T08:51:28.460 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T08:51:28.462 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-03-09T08:51:28.514 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-03-09T08:51:28.516 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:28.538 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:28 +0000 (0:00:00.095) 0:01:23.560 ********* 2021-03-09T08:51:28.608 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-09T08:51:28.609 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:28.631 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:28 +0000 (0:00:00.092) 0:01:23.653 ********* 2021-03-09T08:51:28.703 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T08:51:28.705 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:28.728 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:28 +0000 (0:00:00.096) 0:01:23.749 ********* 2021-03-09T08:51:28.745 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T08:51:28.747 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-03-09T08:51:28.778 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:28 +0000 (0:00:00.050) 0:01:23.799 ********* 2021-03-09T08:51:28.817 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-09T08:51:28.819 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:28.845 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:28 +0000 (0:00:00.067) 0:01:23.867 ********* 2021-03-09T08:51:28.864 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-09T08:51:28.865 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-03-09T08:51:28.892 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:28 +0000 (0:00:00.047) 0:01:23.914 ********* 2021-03-09T08:51:28.920 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-09T08:51:28.922 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:28.950 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:28 +0000 (0:00:00.057) 0:01:23.971 ********* 2021-03-09T08:51:28.988 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-09T08:51:28.989 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:29.017 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.067) 0:01:24.039 ********* 2021-03-09T08:51:29.025 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-09T08:51:29.027 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:29.059 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.042) 0:01:24.081 ********* 2021-03-09T08:51:29.099 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-09T08:51:29.101 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:51:29.137 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.077) 0:01:24.159 ********* 2021-03-09T08:51:29.149 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-09T08:51:29.151 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:29.177 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.039) 0:01:24.199 ********* 2021-03-09T08:51:29.226 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-09T08:51:29.228 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:29.237 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:29.239 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:29.249 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:29.250 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:29.277 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.099) 0:01:24.299 ********* 2021-03-09T08:51:29.301 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.024) 0:01:24.323 ********* 2021-03-09T08:51:29.341 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-03-09T08:51:29.367 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.065) 0:01:24.389 ********* 2021-03-09T08:51:29.418 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-09T08:51:29.420 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:29.446 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.079) 0:01:24.468 ********* 2021-03-09T08:51:29.465 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-09T08:51:29.492 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.045) 0:01:24.514 ********* 2021-03-09T08:51:29.584 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-03-09T08:51:29.608 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.115) 0:01:24.630 ********* 2021-03-09T08:51:29.711 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-03-09T08:51:29.713 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:29.735 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.127) 0:01:24.757 ********* 2021-03-09T08:51:29.837 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-03-09T08:51:29.839 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:29.848 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:29.873 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.138) 0:01:24.895 ********* 2021-03-09T08:51:29.969 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:29.999 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:29 +0000 (0:00:00.125) 0:01:25.021 ********* 2021-03-09T08:51:30.091 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:30.120 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.120) 0:01:25.142 ********* 2021-03-09T08:51:30.138 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-03-09T08:51:30.140 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1615279794, "cache_updated": false, "changed": false} 2021-03-09T08:51:30.170 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.049) 0:01:25.191 ********* 2021-03-09T08:51:30.244 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-03-09T08:51:30.245 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:30.271 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.100) 0:01:25.292 ********* 2021-03-09T08:51:30.351 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-03-09T08:51:30.353 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:30.379 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.108) 0:01:25.401 ********* 2021-03-09T08:51:30.470 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:30.493 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.114) 0:01:25.515 ********* 2021-03-09T08:51:30.565 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:30.589 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.095) 0:01:25.610 ********* 2021-03-09T08:51:30.596 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-03-09T08:51:30.598 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-03-09T08:51:30.624 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.035) 0:01:25.646 ********* 2021-03-09T08:51:30.663 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-09T08:51:30.665 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:30.688 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.063) 0:01:25.710 ********* 2021-03-09T08:51:30.779 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-09T08:51:30.781 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:30.803 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.114) 0:01:25.824 ********* 2021-03-09T08:51:30.883 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T08:51:30.884 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:30.908 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:30 +0000 (0:00:00.104) 0:01:25.929 ********* 2021-03-09T08:51:30.997 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-03-09T08:51:30.999 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:31.022 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.115) 0:01:26.044 ********* 2021-03-09T08:51:31.063 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-03-09T08:51:31.065 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-03-09T08:51:31.092 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.069) 0:01:26.113 ********* 2021-03-09T08:51:31.110 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:31.151 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.057) 0:01:26.171 ********* 2021-03-09T08:51:31.234 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T08:51:31.237 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:31.262 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.109) 0:01:26.281 ********* 2021-03-09T08:51:31.308 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-09T08:51:31.310 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-03-09T08:51:31.331 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:31.357 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.097) 0:01:26.379 ********* 2021-03-09T08:51:31.417 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-09T08:51:31.419 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-03-09T08:51:31.448 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.091) 0:01:26.470 ********* 2021-03-09T08:51:31.458 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-03-09T08:51:31.460 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:31.485 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.036) 0:01:26.507 ********* 2021-03-09T08:51:31.536 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:31.562 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.076) 0:01:26.584 ********* 2021-03-09T08:51:31.570 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-03-09T08:51:31.572 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:31.597 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.035) 0:01:26.619 ********* 2021-03-09T08:51:31.657 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-03-09T08:51:31.660 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:51:31.685 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.088) 0:01:26.707 ********* 2021-03-09T08:51:31.704 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:31.732 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.046) 0:01:26.754 ********* 2021-03-09T08:51:31.734 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:31.772 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-03-09T08:51:31.773 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:31.798 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.065) 0:01:26.820 ********* 2021-03-09T08:51:31.816 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-03-09T08:51:31.817 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:31.842 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.044) 0:01:26.864 ********* 2021-03-09T08:51:31.934 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-03-09T08:51:31.935 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:31.960 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:31 +0000 (0:00:00.117) 0:01:26.982 ********* 2021-03-09T08:51:32.035 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-03-09T08:51:32.036 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:32.064 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:32 +0000 (0:00:00.104) 0:01:27.086 ********* 2021-03-09T08:51:32.144 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-03-09T08:51:32.146 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:32.170 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:32 +0000 (0:00:00.106) 0:01:27.192 ********* 2021-03-09T08:51:32.200 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-03-09T08:51:32.201 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-03-09T08:51:32.227 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:32 +0000 (0:00:00.056) 0:01:27.248 ********* 2021-03-09T08:51:32.371 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-03-09T08:51:32.373 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005398", "end": "2021-03-09 08:51:32.035815", "rc": 0, "start": "2021-03-09 08:51:32.030417", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:32.398 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:32 +0000 (0:00:00.171) 0:01:27.420 ********* 2021-03-09T08:51:32.482 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-09T08:51:32.507 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:32 +0000 (0:00:00.108) 0:01:27.528 ********* 2021-03-09T08:51:32.517 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-03-09T08:51:32.519 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 08:49:09 UTC", "ActiveEnterTimestampMonotonic": "14820229", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target local-fs.target remote-fs.target systemd-tmpfiles-setup.service network.target sysinit.target -.mount systemd-journald.socket time-sync.target system.slice var-run.mount basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 08:49:09 UTC", "AssertTimestampMonotonic": "14783366", "Before": "shutdown.target xdm.service plymouth-quit.service getty@tty1.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 08:49:09 UTC", "Co 2021-03-09T08:51:32.521 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "14783366", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1130", "ExecMainStartTimestamp": "Tue 2021-03-09 08:49:09 UTC", "ExecMainStartTimestampMonotonic": "14820018", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPostEx": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nagios/nrpe.pid ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes" 2021-03-09T08:51:32.522 INFO:teuthology.task.ansible.out:, "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 08:49:09 UTC", "InactiveExitTimestampMonotonic": "14820229", "InvocationID": "7ac16f60b7bb42f8b30ec9f704e9de16", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127781", "LimitNPROCSoft": "127781", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127781", "LimitSIGPENDINGSoft": "127781", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1130", "MemoryAccounting": "yes", "MemoryCurrent": "1441792", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", 2021-03-09T08:51:32.524 INFO:teuthology.task.ansible.out:"MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nagios/nrpe.pid", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:49:09 UTC", "StateChangeTimestampMonotonic": "14820229", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", 2021-03-09T08:51:32.526 INFO:teuthology.task.ansible.out:"SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "38334", "TimeoutAbortUSec": "1min", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "110", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-09T08:51:32.548 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:32 +0000 (0:00:00.040) 0:01:27.569 ********* 2021-03-09T08:51:32.650 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:32.674 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:32 +0000 (0:00:00.126) 0:01:27.696 ********* 2021-03-09T08:51:32.755 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:32.785 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:32 +0000 (0:00:00.109) 0:01:27.806 ********* 2021-03-09T08:51:32.866 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-03-09T08:51:32.868 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:32.895 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:32 +0000 (0:00:00.111) 0:01:27.917 ********* 2021-03-09T08:51:32.903 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-03-09T08:51:32.905 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-03-09T08:51:32.941 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:32 +0000 (0:00:00.045) 0:01:27.963 ********* 2021-03-09T08:51:32.983 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-03-09T08:51:32.985 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:33.012 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:33 +0000 (0:00:00.070) 0:01:28.033 ********* 2021-03-09T08:51:33.029 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-09T08:51:33.031 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:33.069 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:33 +0000 (0:00:00.056) 0:01:28.090 ********* 2021-03-09T08:51:33.091 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-03-09T08:51:33.093 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:33.117 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:33 +0000 (0:00:00.048) 0:01:28.139 ********* 2021-03-09T08:51:33.218 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:33.243 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:33 +0000 (0:00:00.125) 0:01:28.265 ********* 2021-03-09T08:51:33.338 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:33.362 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:33 +0000 (0:00:00.118) 0:01:28.383 ********* 2021-03-09T08:51:33.441 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-03-09T08:51:33.442 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:33.465 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:33 +0000 (0:00:00.103) 0:01:28.487 ********* 2021-03-09T08:51:33.544 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-03-09T08:51:33.546 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:33.569 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:33 +0000 (0:00:00.103) 0:01:28.591 ********* 2021-03-09T08:51:33.648 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-03-09T08:51:33.650 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:33.672 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:33 +0000 (0:00:00.102) 0:01:28.693 ********* 2021-03-09T08:51:33.751 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-03-09T08:51:33.753 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-03-09T08:51:33.782 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:33 +0000 (0:00:00.110) 0:01:28.804 ********* 2021-03-09T08:51:33.874 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-03-09T08:51:33.876 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-03-09T08:51:33.901 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:33 +0000 (0:00:00.118) 0:01:28.922 ********* 2021-03-09T08:51:33.992 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-03-09T08:51:33.994 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "docker.io", "python3-nose", "python3-virtualenv", "python3-configobj", "python3-gevent", "python3-numpy", "python3-matplotlib", "python3-setuptools"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml"} 2021-03-09T08:51:34.016 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:34 +0000 (0:00:00.115) 0:01:29.038 ********* 2021-03-09T08:51:34.097 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-09T08:51:34.098 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-09T08:51:34.124 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:34 +0000 (0:00:00.107) 0:01:29.145 ********* 2021-03-09T08:51:34.257 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-03-09T08:51:34.259 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-09T08:51:34.281 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:34 +0000 (0:00:00.157) 0:01:29.303 ********* 2021-03-09T08:51:34.470 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-03-09T08:51:34.472 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-09T08:51:34.500 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:34 +0000 (0:00:00.218) 0:01:29.522 ********* 2021-03-09T08:51:34.507 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-09T08:51:34.514 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-03-09T08:51:34.593 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:34.618 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:34 +0000 (0:00:00.117) 0:01:29.639 ********* 2021-03-09T08:51:34.689 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:34.716 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:34 +0000 (0:00:00.098) 0:01:29.738 ********* 2021-03-09T08:51:34.940 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi146.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-03-09T08:51:34.942 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:34.966 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:34 +0000 (0:00:00.249) 0:01:29.988 ********* 2021-03-09T08:51:35.440 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-09T08:51:35.441 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-03-09T08:51:35.469 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:35 +0000 (0:00:00.502) 0:01:30.490 ********* 2021-03-09T08:51:35.489 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-09T08:51:35.491 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-03-09T08:51:35.520 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:35 +0000 (0:00:00.050) 0:01:30.541 ********* 2021-03-09T08:51:35.551 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi146"}, "changed": false} 2021-03-09T08:51:35.577 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:35 +0000 (0:00:00.057) 0:01:30.599 ********* 2021-03-09T08:51:36.075 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-03-09T08:51:36.077 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi146", "ansible_hostname": "smithi146", "ansible_nodename": "smithi146"}, "changed": false, "name": "smithi146"} 2021-03-09T08:51:36.101 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:36 +0000 (0:00:00.523) 0:01:31.122 ********* 2021-03-09T08:51:36.256 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-03-09T08:51:36.258 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"cache_update_time": 1615279787, "cache_updated": false, "changed": false} 2021-03-09T08:51:36.286 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:36 +0000 (0:00:00.185) 0:01:31.308 ********* 2021-03-09T08:51:36.368 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-09T08:51:36.393 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:36 +0000 (0:00:00.106) 0:01:31.415 ********* 2021-03-09T08:51:36.485 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-03-09T08:51:36.511 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:36 +0000 (0:00:00.117) 0:01:31.532 ********* 2021-03-09T08:51:36.613 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-03-09T08:51:36.615 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.033490", "end": "2021-03-09 08:51:36.288320", "rc": 0, "start": "2021-03-09 08:51:36.254830", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:36.642 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:36 +0000 (0:00:00.131) 0:01:31.664 ********* 2021-03-09T08:51:36.933 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-03-09T08:51:36.935 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-03-09T08:51:36.959 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:36 +0000 (0:00:00.317) 0:01:31.981 ********* 2021-03-09T08:51:37.433 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:37.435 INFO:teuthology.task.ansible.out:TASK [testnode : Add ssh pubkeys] ********************************************** changed: [smithi146.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-09T08:51:37.458 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:37 +0000 (0:00:00.498) 0:01:32.479 ********* 2021-03-09T08:51:37.540 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-03-09T08:51:37.542 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:37.573 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:37 +0000 (0:00:00.115) 0:01:32.595 ********* 2021-03-09T08:51:37.655 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-09T08:51:37.657 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:37.680 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:37 +0000 (0:00:00.106) 0:01:32.701 ********* 2021-03-09T08:51:37.772 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T08:51:37.774 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:37.798 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:37 +0000 (0:00:00.118) 0:01:32.820 ********* 2021-03-09T08:51:37.902 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:37.926 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:37 +0000 (0:00:00.128) 0:01:32.948 ********* 2021-03-09T08:51:38.018 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:38.047 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:38 +0000 (0:00:00.120) 0:01:33.068 ********* 2021-03-09T08:51:38.116 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-09T08:51:38.118 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:38.140 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:38 +0000 (0:00:00.093) 0:01:33.162 ********* 2021-03-09T08:51:38.210 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-09T08:51:38.212 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:38.235 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:38 +0000 (0:00:00.094) 0:01:33.257 ********* 2021-03-09T08:51:38.304 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-03-09T08:51:38.307 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:38.328 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:38 +0000 (0:00:00.092) 0:01:33.350 ********* 2021-03-09T08:51:38.398 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-03-09T08:51:38.424 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:38 +0000 (0:00:00.095) 0:01:33.445 ********* 2021-03-09T08:51:38.504 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-09T08:51:38.529 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:38 +0000 (0:00:00.105) 0:01:33.551 ********* 2021-03-09T08:51:38.601 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-03-09T08:51:38.625 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:38 +0000 (0:00:00.096) 0:01:33.647 ********* 2021-03-09T08:51:38.705 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-03-09T08:51:38.706 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:38.731 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:38 +0000 (0:00:00.106) 0:01:33.753 ********* 2021-03-09T08:51:38.844 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-03-09T08:51:38.845 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-09T08:51:38.869 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:38 +0000 (0:00:00.138) 0:01:33.891 ********* 2021-03-09T08:51:38.952 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-03-09T08:51:38.954 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"cache_update_time": 1615279787, "cache_updated": false, "changed": false} TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:38.980 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:38 +0000 (0:00:00.109) 0:01:34.001 ********* 2021-03-09T08:51:39.005 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:38 +0000 (0:00:00.025) 0:01:34.026 ********* 2021-03-09T08:51:39.114 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:51:39.116 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:39.138 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:39 +0000 (0:00:00.133) 0:01:34.160 ********* 2021-03-09T08:51:39.217 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:39.219 INFO:teuthology.task.ansible.out:TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:39.246 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:39 +0000 (0:00:00.107) 0:01:34.267 ********* 2021-03-09T08:51:39.319 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-03-09T08:51:39.321 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:39.345 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:39 +0000 (0:00:00.098) 0:01:34.366 ********* 2021-03-09T08:51:39.415 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:39.417 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:39.438 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:39 +0000 (0:00:00.093) 0:01:34.460 ********* 2021-03-09T08:51:39.509 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-03-09T08:51:39.511 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:39.533 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:39 +0000 (0:00:00.094) 0:01:34.555 ********* 2021-03-09T08:51:39.603 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-09T08:51:39.604 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:39.627 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:39 +0000 (0:00:00.093) 0:01:34.649 ********* 2021-03-09T08:51:39.697 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-09T08:51:39.699 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:39.721 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:39 +0000 (0:00:00.093) 0:01:34.743 ********* 2021-03-09T08:51:39.800 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T08:51:39.802 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:39.824 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:39 +0000 (0:00:00.102) 0:01:34.846 ********* 2021-03-09T08:51:39.926 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-03-09T08:51:39.928 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:39.950 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:39 +0000 (0:00:00.126) 0:01:34.972 ********* 2021-03-09T08:51:40.020 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-09T08:51:40.022 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:40.045 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:40 +0000 (0:00:00.095) 0:01:35.067 ********* 2021-03-09T08:51:40.125 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T08:51:40.127 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:40.149 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:40 +0000 (0:00:00.103) 0:01:35.171 ********* 2021-03-09T08:51:40.241 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-03-09T08:51:40.243 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:40.265 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:40 +0000 (0:00:00.115) 0:01:35.287 ********* 2021-03-09T08:51:40.345 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-03-09T08:51:40.347 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:40.368 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:40 +0000 (0:00:00.103) 0:01:35.390 ********* 2021-03-09T08:51:40.448 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-03-09T08:51:40.450 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:40.478 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:40 +0000 (0:00:00.109) 0:01:35.500 ********* 2021-03-09T08:51:40.547 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-03-09T08:51:40.548 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:40.550 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:40.574 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:40 +0000 (0:00:00.095) 0:01:35.595 ********* 2021-03-09T08:51:40.576 INFO:teuthology.task.ansible.out: 2021-03-09T08:51:40.676 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-03-09T08:51:40.678 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:40.703 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:40 +0000 (0:00:00.128) 0:01:35.724 ********* 2021-03-09T08:51:40.783 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-03-09T08:51:40.785 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:40.809 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:40 +0000 (0:00:00.106) 0:01:35.831 ********* 2021-03-09T08:51:40.880 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-03-09T08:51:40.882 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:40.905 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:40 +0000 (0:00:00.096) 0:01:35.927 ********* 2021-03-09T08:51:40.985 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-03-09T08:51:40.987 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:41.009 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:41 +0000 (0:00:00.103) 0:01:36.031 ********* 2021-03-09T08:51:41.215 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-03-09T08:51:41.217 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005388", "end": "2021-03-09 08:51:41.618281", "rc": 0, "start": "2021-03-09 08:51:41.612893", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:41.241 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:41 +0000 (0:00:00.231) 0:01:36.263 ********* 2021-03-09T08:51:41.321 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-09T08:51:41.345 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:41 +0000 (0:00:00.104) 0:01:36.367 ********* 2021-03-09T08:51:41.740 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-03-09T08:51:41.742 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-03-09T08:51:41.765 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:41 +0000 (0:00:00.419) 0:01:36.787 ********* 2021-03-09T08:51:41.845 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-09T08:51:41.847 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:41.871 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:41 +0000 (0:00:00.105) 0:01:36.892 ********* 2021-03-09T08:51:42.952 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-09T08:51:42.953 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-03-09T08:51:43.911 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-03-09T08:51:43.935 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:43 +0000 (0:00:02.064) 0:01:38.957 ********* 2021-03-09T08:51:44.632 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-03-09T08:51:44.634 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1615279794, "cache_updated": false, "changed": false} 2021-03-09T08:51:44.658 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:44 +0000 (0:00:00.723) 0:01:39.680 ********* 2021-03-09T08:51:44.729 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-09T08:51:44.752 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:44 +0000 (0:00:00.094) 0:01:39.774 ********* 2021-03-09T08:51:44.979 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-03-09T08:51:44.981 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.033055", "end": "2021-03-09 08:51:45.375333", "rc": 0, "start": "2021-03-09 08:51:45.342278", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:51:45.008 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:45 +0000 (0:00:00.255) 0:01:40.030 ********* 2021-03-09T08:51:47.261 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-03-09T08:51:47.263 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1615279794, "cache_updated": false, "changed": false} 2021-03-09T08:51:47.285 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:47 +0000 (0:00:02.276) 0:01:42.307 ********* 2021-03-09T08:51:49.700 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-03-09T08:51:49.702 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1\n libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0\n libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0\n libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0\n libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla\n libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0\n libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf\n qemu-system-data qemu-system-gui seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 7 not upgraded.\nAfter this operation, 62.7 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101691 files and directories currently installed.)\r\n 2021-03-09T08:51:49.704 INFO:teuthology.task.ansible.out:Removing qemu-utils (1:4.2-3ubuntu6.14) ...\r\nRemoving qemu-system-x86 (1:4.2-3ubuntu6.14) ...\r\nRemoving qemu-system-common (1:4.2-3ubuntu6.14) ...\r\nRemoving qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...\r\nRemoving librbd1 (15.2.8-0ubuntu0.20.04.1) ...\r\nRemoving librados2 (15.2.8-0ubuntu0.20.04.1) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1", " libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0", " libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0", " libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0", " libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla", " libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0", " libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf", " qemu-system-data qemu-system-gui seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 7 not upgraded.", "After this operation, 62.7 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Readin 2021-03-09T08:51:49.706 INFO:teuthology.task.ansible.out:g database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101691 files and directories currently installed.)", "Removing qemu-utils (1:4.2-3ubuntu6.14) ...", "Removing qemu-system-x86 (1:4.2-3ubuntu6.14) ...", "Removing qemu-system-common (1:4.2-3ubuntu6.14) ...", "Removing qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...", "Removing librbd1 (15.2.8-0ubuntu0.20.04.1) ...", "Removing librados2 (15.2.8-0ubuntu0.20.04.1) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ..."]} 2021-03-09T08:51:49.727 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:49 +0000 (0:00:02.441) 0:01:44.749 ********* 2021-03-09T08:51:50.438 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* ok: [smithi005.front.sepia.ceph.com] => {"cache_update_time": 1615279787, "cache_updated": false, "changed": false} 2021-03-09T08:51:50.462 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:50 +0000 (0:00:00.735) 0:01:45.484 ********* 2021-03-09T08:51:50.544 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-09T08:51:50.546 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:50.570 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:50 +0000 (0:00:00.107) 0:01:45.592 ********* 2021-03-09T08:51:57.677 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-03-09T08:51:57.679 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1\n libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0\n libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0\n libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0\n libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla\n libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0\n libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf\n qemu-system-data qemu-system-gui seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 7 not upgraded.\nAfter this operation, 62.7 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101691 files and directories currently installed.)\r\n 2021-03-09T08:51:57.681 INFO:teuthology.task.ansible.out:Removing qemu-utils (1:4.2-3ubuntu6.14) ...\r\nRemoving qemu-system-x86 (1:4.2-3ubuntu6.14) ...\r\nRemoving qemu-system-common (1:4.2-3ubuntu6.14) ...\r\nRemoving qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...\r\nRemoving librbd1 (15.2.8-0ubuntu0.20.04.1) ...\r\nRemoving librados2 (15.2.8-0ubuntu0.20.04.1) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1", " libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0", " libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0", " libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0", " libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla", " libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0", " libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf", " qemu-system-data qemu-system-gui seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 7 not upgraded.", "After this operation, 62.7 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Readin 2021-03-09T08:51:57.682 INFO:teuthology.task.ansible.out:g database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101691 files and directories currently installed.)", "Removing qemu-utils (1:4.2-3ubuntu6.14) ...", "Removing qemu-system-x86 (1:4.2-3ubuntu6.14) ...", "Removing qemu-system-common (1:4.2-3ubuntu6.14) ...", "Removing qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...", "Removing librbd1 (15.2.8-0ubuntu0.20.04.1) ...", "Removing librados2 (15.2.8-0ubuntu0.20.04.1) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ..."]} 2021-03-09T08:51:57.716 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:57 +0000 (0:00:07.146) 0:01:52.738 ********* 2021-03-09T08:51:58.438 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T08:51:58.440 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1615279794, "cache_updated": false, "changed": false} 2021-03-09T08:51:58.462 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:58 +0000 (0:00:00.746) 0:01:53.484 ********* 2021-03-09T08:51:58.534 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-09T08:51:58.536 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:51:58.561 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:51:58 +0000 (0:00:00.098) 0:01:53.582 ********* 2021-03-09T08:52:00.314 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T08:52:00.316 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"cache_update_time": 1615279787, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.25.1-1ubuntu3).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 7 not upgraded.\nNeed to get 13.6 MB of archives.\nAfter this operation, 62.7 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.8-0ubuntu0.20.04.1 [3209 kB]\nGet:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.8-0ubuntu0.20.04.1 [1615 kB]\nGet:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.14 [53.1 kB]\nGet:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.14 [1055 kB]\nGet:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.14 [6726 kB]\nGet:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.14 [975 kB]\nFetched 13.6 MB in 1s (9301 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading dat 2021-03-09T08:52:00.318 INFO:teuthology.task.ansible.out:abase ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101619 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_15.2.8-0ubuntu0.20.04.1_amd64.deb ...\r\nUnpacking librados2 (15.2.8-0ubuntu0.20.04.1) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_15.2.8-0ubuntu0.20.04.1_amd64.deb ...\r\nUnpacking librbd1 (15.2.8-0ubuntu0.20.04.1) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a4.2-3ubuntu6.14_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a4.2-3ubuntu6.14_amd64.deb ...\r\nUnpacking qemu-system-common (1:4.2-3ubuntu6.14) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a4.2-3ubuntu6.14_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:4.2-3ubuntu6.14) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a4.2-3ubuntu6.14_amd64.deb ...\r\nUnpacking qemu-utils (1:4.2-3ubuntu6.14) ...\r\nSetting up librados2 (15.2.8-0ubuntu0.20.04.1) ...\r\nSetting up librbd1 (15.2.8-0ubuntu0.20.04.1) ...\r\nSetting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...\r\nSetting up qemu-system-common (1:4.2-3ubuntu6.14) ...\r\nSetting up qemu-system-x86 (1:4.2-3ubuntu6.14) ...\r\nSetting up qemu-utils (1:4.2-3ubuntu6.14) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is a 2021-03-09T08:52:00.319 INFO:teuthology.task.ansible.out:lready the newest version (1:2.25.1-1ubuntu3).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 7 not upgraded.", "Need to get 13.6 MB of archives.", "After this operation, 62.7 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.8-0ubuntu0.20.04.1 [3209 kB]", "Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.8-0ubuntu0.20.04.1 [1615 kB]", "Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.14 [53.1 kB]", "Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.14 [1055 kB]", "Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.14 [6726 kB]", "Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.14 [975 kB]", "Fetched 13.6 MB in 1s (9301 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101619 files and directories currently ins 2021-03-09T08:52:00.321 INFO:teuthology.task.ansible.out:talled.)", "Preparing to unpack .../0-librados2_15.2.8-0ubuntu0.20.04.1_amd64.deb ...", "Unpacking librados2 (15.2.8-0ubuntu0.20.04.1) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_15.2.8-0ubuntu0.20.04.1_amd64.deb ...", "Unpacking librbd1 (15.2.8-0ubuntu0.20.04.1) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a4.2-3ubuntu6.14_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a4.2-3ubuntu6.14_amd64.deb ...", "Unpacking qemu-system-common (1:4.2-3ubuntu6.14) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a4.2-3ubuntu6.14_amd64.deb ...", "Unpacking qemu-system-x86 (1:4.2-3ubuntu6.14) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a4.2-3ubuntu6.14_amd64.deb ...", "Unpacking qemu-utils (1:4.2-3ubuntu6.14) ...", "Setting up librados2 (15.2.8-0ubuntu0.20.04.1) ...", "Setting up librbd1 (15.2.8-0ubuntu0.20.04.1) ...", "Setting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...", "Setting up qemu-system-common (1:4.2-3ubuntu6.14) ...", "Setting up qemu-system-x86 (1:4.2-3ubuntu6.14) ...", "Setting up qemu-utils (1:4.2-3ubuntu6.14) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ..."]} 2021-03-09T08:52:00.354 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:00 +0000 (0:00:01.793) 0:01:55.376 ********* 2021-03-09T08:52:01.052 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-03-09T08:52:01.054 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"cache_update_time": 1615279787, "cache_updated": false, "changed": false} 2021-03-09T08:52:01.077 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:01 +0000 (0:00:00.723) 0:01:56.099 ********* 2021-03-09T08:52:01.784 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-03-09T08:52:01.786 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"cache_update_time": 1615279787, "cache_updated": false, "changed": false} 2021-03-09T08:52:01.809 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:01 +0000 (0:00:00.732) 0:01:56.831 ********* 2021-03-09T08:52:02.006 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-09T08:52:02.008 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-03-09T08:52:02.034 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:02 +0000 (0:00:00.224) 0:01:57.056 ********* 2021-03-09T08:52:02.115 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-09T08:52:02.117 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:02.139 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:02 +0000 (0:00:00.105) 0:01:57.161 ********* 2021-03-09T08:52:02.324 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-09T08:52:02.326 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-03-09T08:52:02.349 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:02 +0000 (0:00:00.209) 0:01:57.370 ********* 2021-03-09T08:52:02.754 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-09T08:52:02.757 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-03-09T08:52:02.778 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:02 +0000 (0:00:00.428) 0:01:57.799 ********* 2021-03-09T08:52:03.010 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-03-09T08:52:03.012 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-09T08:52:03.034 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:03 +0000 (0:00:00.256) 0:01:58.056 ********* 2021-03-09T08:52:03.242 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-09T08:52:03.244 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:52:03.266 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:03 +0000 (0:00:00.231) 0:01:58.288 ********* 2021-03-09T08:52:03.595 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 08:48:21 UTC", "ActiveEnterTimestampMonotonic": "14900621", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target system.slice systemd-tmpfiles-setup.service -.mount sysinit.target remote-fs.target nss-lookup.target systemd-journald.socket basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 08:48:19 UTC", "AssertTimestampMonotonic": "13452632", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 08:48:19 UTC", "ConditionTimestampMonotonic": "13452632", 2021-03-09T08:52:03.597 INFO:teuthology.task.ansible.out:"ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", "Documentation": "https://httpd.apache.org/docs/2.4/", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "860", "ExecMainStartTimestamp": "Tue 2021-03-09 08:48:21 UTC", "ExecMainStartTimestampMonotonic": "14900598", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", " 2021-03-09T08:52:03.599 INFO:teuthology.task.ansible.out:IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 08:48:19 UTC", "InactiveExitTimestampMonotonic": "13453815", "InvocationID": "29eb26d86c364ac7be676cbe1a1f9832", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127805", "LimitNPROCSoft": "127805", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127805", "LimitSIGPENDINGSoft": "127805", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "860", "MemoryAccounting": "yes", "MemoryCurrent": "10088448", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountF 2021-03-09T08:52:03.600 INFO:teuthology.task.ansible.out:lags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:48:21 UTC", "StateChangeTimestampMonotonic": "14900621", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "Task 2021-03-09T08:52:03.602 INFO:teuthology.task.ansible.out:sAccounting": "yes", "TasksCurrent": "55", "TasksMax": "38341", "TimeoutAbortUSec": "1min 30s", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-09T08:52:03.622 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:03 +0000 (0:00:00.355) 0:01:58.643 ********* 2021-03-09T08:52:03.703 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-09T08:52:03.704 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:03.726 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:03 +0000 (0:00:00.104) 0:01:58.748 ********* 2021-03-09T08:52:03.810 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T08:52:03.812 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:03.834 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:03 +0000 (0:00:00.107) 0:01:58.856 ********* 2021-03-09T08:52:03.904 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-09T08:52:03.905 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:03.928 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:03 +0000 (0:00:00.093) 0:01:58.950 ********* 2021-03-09T08:52:04.012 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-09T08:52:04.014 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:04.036 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:04 +0000 (0:00:00.107) 0:01:59.058 ********* 2021-03-09T08:52:04.116 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:04.140 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:04 +0000 (0:00:00.103) 0:01:59.161 ********* 2021-03-09T08:52:04.230 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-09T08:52:04.232 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:04.254 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:04 +0000 (0:00:00.114) 0:01:59.276 ********* 2021-03-09T08:52:04.334 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-09T08:52:04.336 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:04.362 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:04 +0000 (0:00:00.107) 0:01:59.384 ********* 2021-03-09T08:52:04.432 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:04.457 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:04 +0000 (0:00:00.094) 0:01:59.479 ********* 2021-03-09T08:52:04.525 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:52:04.527 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:04.550 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:04 +0000 (0:00:00.093) 0:01:59.572 ********* 2021-03-09T08:52:04.620 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-09T08:52:04.622 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:04.648 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:04 +0000 (0:00:00.097) 0:01:59.669 ********* 2021-03-09T08:52:04.718 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T08:52:04.719 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:04.742 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:04 +0000 (0:00:00.094) 0:01:59.764 ********* 2021-03-09T08:52:04.833 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T08:52:04.835 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:04.858 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:04 +0000 (0:00:00.115) 0:01:59.880 ********* 2021-03-09T08:52:04.930 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-09T08:52:04.931 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:04.955 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:04 +0000 (0:00:00.097) 0:01:59.977 ********* 2021-03-09T08:52:05.036 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T08:52:05.038 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:05.060 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:05 +0000 (0:00:00.104) 0:02:00.082 ********* 2021-03-09T08:52:05.151 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T08:52:05.153 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:05.176 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:05 +0000 (0:00:00.115) 0:02:00.197 ********* 2021-03-09T08:52:05.257 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T08:52:05.259 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:05.284 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:05 +0000 (0:00:00.108) 0:02:00.306 ********* 2021-03-09T08:52:05.366 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:05.368 INFO:teuthology.task.ansible.out:TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:05.391 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:05 +0000 (0:00:00.106) 0:02:00.412 ********* 2021-03-09T08:52:05.514 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T08:52:05.517 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:05.538 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:05 +0000 (0:00:00.147) 0:02:00.560 ********* 2021-03-09T08:52:05.620 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-09T08:52:05.624 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:05.649 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:05 +0000 (0:00:00.109) 0:02:00.670 ********* 2021-03-09T08:52:05.762 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T08:52:05.764 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:05.797 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:05 +0000 (0:00:00.149) 0:02:00.819 ********* 2021-03-09T08:52:05.913 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T08:52:05.915 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:05.940 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:05 +0000 (0:00:00.142) 0:02:00.962 ********* 2021-03-09T08:52:06.036 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T08:52:06.037 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:06.068 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:06 +0000 (0:00:00.128) 0:02:01.090 ********* 2021-03-09T08:52:06.151 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T08:52:06.154 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:06.177 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:06 +0000 (0:00:00.109) 0:02:01.199 ********* 2021-03-09T08:52:06.271 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T08:52:06.273 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:06.304 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:06 +0000 (0:00:00.126) 0:02:01.325 ********* 2021-03-09T08:52:06.388 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T08:52:06.389 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:06.413 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:06 +0000 (0:00:00.109) 0:02:01.435 ********* 2021-03-09T08:52:06.484 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-03-09T08:52:06.486 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:06.509 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:06 +0000 (0:00:00.095) 0:02:01.530 ********* 2021-03-09T08:52:06.715 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-03-09T08:52:06.718 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-03-09T08:52:06.748 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:06 +0000 (0:00:00.239) 0:02:01.770 ********* 2021-03-09T08:52:07.162 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T08:52:07.164 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-03-09T08:52:07.186 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:07 +0000 (0:00:00.437) 0:02:02.208 ********* 2021-03-09T08:52:07.266 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T08:52:07.268 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:07.291 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:07 +0000 (0:00:00.105) 0:02:02.313 ********* 2021-03-09T08:52:07.691 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-03-09T08:52:07.693 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-03-09T08:52:07.716 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:07 +0000 (0:00:00.424) 0:02:02.737 ********* 2021-03-09T08:52:08.124 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-03-09T08:52:08.126 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-03-09T08:52:08.166 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:08 +0000 (0:00:00.450) 0:02:03.188 ********* 2021-03-09T08:52:08.221 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T08:52:08.223 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1615279794, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.25.1-1ubuntu3).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 7 not upgraded.\nNeed to get 13.6 MB of archives.\nAfter this operation, 62.7 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.8-0ubuntu0.20.04.1 [3209 kB]\nGet:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.8-0ubuntu0.20.04.1 [1615 kB]\nGet:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.14 [53.1 kB]\nGet:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.14 [1055 kB]\nGet:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.14 [6726 kB]\nGet:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.14 [975 kB]\nFetched 13.6 MB in 1s (10.3 MB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading dat 2021-03-09T08:52:08.225 INFO:teuthology.task.ansible.out:abase ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101619 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_15.2.8-0ubuntu0.20.04.1_amd64.deb ...\r\nUnpacking librados2 (15.2.8-0ubuntu0.20.04.1) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_15.2.8-0ubuntu0.20.04.1_amd64.deb ...\r\nUnpacking librbd1 (15.2.8-0ubuntu0.20.04.1) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a4.2-3ubuntu6.14_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a4.2-3ubuntu6.14_amd64.deb ...\r\nUnpacking qemu-system-common (1:4.2-3ubuntu6.14) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a4.2-3ubuntu6.14_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:4.2-3ubuntu6.14) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a4.2-3ubuntu6.14_amd64.deb ...\r\nUnpacking qemu-utils (1:4.2-3ubuntu6.14) ...\r\nSetting up librados2 (15.2.8-0ubuntu0.20.04.1) ...\r\nSetting up librbd1 (15.2.8-0ubuntu0.20.04.1) ...\r\nSetting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...\r\nSetting up qemu-system-common (1:4.2-3ubuntu6.14) ...\r\nSetting up qemu-system-x86 (1:4.2-3ubuntu6.14) ...\r\nSetting up qemu-utils (1:4.2-3ubuntu6.14) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is a 2021-03-09T08:52:08.227 INFO:teuthology.task.ansible.out:lready the newest version (1:2.25.1-1ubuntu3).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 7 not upgraded.", "Need to get 13.6 MB of archives.", "After this operation, 62.7 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.8-0ubuntu0.20.04.1 [3209 kB]", "Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.8-0ubuntu0.20.04.1 [1615 kB]", "Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.14 [53.1 kB]", "Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.14 [1055 kB]", "Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.14 [6726 kB]", "Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.14 [975 kB]", "Fetched 13.6 MB in 1s (10.3 MB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101619 files and directories currently ins 2021-03-09T08:52:08.228 INFO:teuthology.task.ansible.out:talled.)", "Preparing to unpack .../0-librados2_15.2.8-0ubuntu0.20.04.1_amd64.deb ...", "Unpacking librados2 (15.2.8-0ubuntu0.20.04.1) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_15.2.8-0ubuntu0.20.04.1_amd64.deb ...", "Unpacking librbd1 (15.2.8-0ubuntu0.20.04.1) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a4.2-3ubuntu6.14_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a4.2-3ubuntu6.14_amd64.deb ...", "Unpacking qemu-system-common (1:4.2-3ubuntu6.14) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a4.2-3ubuntu6.14_amd64.deb ...", "Unpacking qemu-system-x86 (1:4.2-3ubuntu6.14) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a4.2-3ubuntu6.14_amd64.deb ...", "Unpacking qemu-utils (1:4.2-3ubuntu6.14) ...", "Setting up librados2 (15.2.8-0ubuntu0.20.04.1) ...", "Setting up librbd1 (15.2.8-0ubuntu0.20.04.1) ...", "Setting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...", "Setting up qemu-system-common (1:4.2-3ubuntu6.14) ...", "Setting up qemu-system-x86 (1:4.2-3ubuntu6.14) ...", "Setting up qemu-utils (1:4.2-3ubuntu6.14) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ..."]} 2021-03-09T08:52:08.259 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:08 +0000 (0:00:00.092) 0:02:03.281 ********* 2021-03-09T08:52:08.632 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-03-09T08:52:08.634 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-03-09T08:52:08.667 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:08 +0000 (0:00:00.408) 0:02:03.689 ********* 2021-03-09T08:52:08.863 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-03-09T08:52:08.865 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:52:08.896 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:08 +0000 (0:00:00.228) 0:02:03.918 ********* 2021-03-09T08:52:08.988 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-03-09T08:52:08.990 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Install non aarch64 packages] ********************************* ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1615279794, "cache_updated": false, "changed": false} 2021-03-09T08:52:09.016 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:09 +0000 (0:00:00.119) 0:02:04.038 ********* 2021-03-09T08:52:09.043 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:09 +0000 (0:00:00.027) 0:02:04.065 ********* 2021-03-09T08:52:09.098 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-03-09T08:52:09.099 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.101 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:09.137 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:09 +0000 (0:00:00.093) 0:02:04.159 ********* 2021-03-09T08:52:09.229 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.264 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:09 +0000 (0:00:00.126) 0:02:04.285 ********* 2021-03-09T08:52:09.348 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.382 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:09 +0000 (0:00:00.117) 0:02:04.403 ********* 2021-03-09T08:52:09.475 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-09T08:52:09.477 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.500 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:09 +0000 (0:00:00.118) 0:02:04.522 ********* 2021-03-09T08:52:09.593 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:52:09.594 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-09T08:52:09.621 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:09 +0000 (0:00:00.120) 0:02:04.643 ********* 2021-03-09T08:52:09.737 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-09T08:52:09.739 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.749 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.750 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.761 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.763 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.773 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-03-09T08:52:09.775 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1615279794, "cache_updated": false, "changed": false} TASK [testnode : Check for NVMe drive] ***************************************** ok: [smithi005.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-03-09T08:52:09.806 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:09 +0000 (0:00:00.185) 0:02:04.828 ********* 2021-03-09T08:52:09.817 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.821 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.822 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.823 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.825 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.826 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.826 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.826 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:09.874 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:09 +0000 (0:00:00.067) 0:02:04.896 ********* 2021-03-09T08:52:09.991 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-09T08:52:09.992 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-03-09T08:52:10.016 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:10 +0000 (0:00:00.141) 0:02:05.038 ********* 2021-03-09T08:52:10.128 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-09T08:52:10.130 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:10.152 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:10 +0000 (0:00:00.135) 0:02:05.173 ********* 2021-03-09T08:52:10.292 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-03-09T08:52:10.294 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.006970", "end": "2021-03-09 08:52:10.267752", "rc": 0, "start": "2021-03-09 08:52:10.260782", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock"]} 2021-03-09T08:52:10.321 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:10 +0000 (0:00:00.169) 0:02:05.342 ********* 2021-03-09T08:52:10.348 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-09T08:52:10.350 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-03-09T08:52:10.375 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:10 +0000 (0:00:00.054) 0:02:05.397 ********* 2021-03-09T08:52:10.402 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-03-09T08:52:10.404 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:10.431 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:10 +0000 (0:00:00.056) 0:02:05.453 ********* 2021-03-09T08:52:10.512 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-03-09T08:52:10.514 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:10.537 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:10 +0000 (0:00:00.105) 0:02:05.559 ********* 2021-03-09T08:52:10.648 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-03-09T08:52:10.650 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:10.674 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:10 +0000 (0:00:00.136) 0:02:05.695 ********* 2021-03-09T08:52:10.885 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-09T08:52:10.887 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-03-09T08:52:10.917 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:10 +0000 (0:00:00.242) 0:02:05.938 ********* 2021-03-09T08:52:11.181 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-03-09T08:52:11.183 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-09T08:52:11.219 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:11 +0000 (0:00:00.302) 0:02:06.241 ********* 2021-03-09T08:52:11.408 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-03-09T08:52:11.410 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"cache_update_time": 1615279787, "cache_updated": false, "changed": false} 2021-03-09T08:52:11.445 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:11 +0000 (0:00:00.225) 0:02:06.467 ********* 2021-03-09T08:52:11.518 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-09T08:52:11.520 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:52:11.546 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:11 +0000 (0:00:00.101) 0:02:06.568 ********* 2021-03-09T08:52:11.554 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-09T08:52:11.556 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:11.602 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:11 +0000 (0:00:00.055) 0:02:06.623 ********* 2021-03-09T08:52:11.734 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T08:52:11.735 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item={'block_used': 13728278, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226366830, 'size_available': 927198535680, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 113650, 'block_size': 4096, 'inode_available': 60941326}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226366830, "block_size": 4096, "block_total": 240095108, "block_used": 13728278, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941326, "inode_total": 61054976, "inode_used": 113650, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 927198535680, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}} 2021-03-09T08:52:11.762 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:11 +0000 (0:00:00.160) 0:02:06.784 ********* 2021-03-09T08:52:11.903 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-03-09T08:52:11.906 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 08:49:09 UTC", "ActiveEnterTimestampMonotonic": "15234054", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target nss-lookup.target basic.target system.slice sysinit.target systemd-journald.socket systemd-tmpfiles-setup.service -.mount network.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 08:49:09 UTC", "AssertTimestampMonotonic": "14703281", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 08:49:09 UTC", "ConditionTimestampMonotonic": "14703281", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup" 2021-03-09T08:52:11.908 INFO:teuthology.task.ansible.out:: "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", "Documentation": "https://httpd.apache.org/docs/2.4/", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1370", "ExecMainStartTimestamp": "Tue 2021-03-09 08:49:09 UTC", "ExecMainStartTimestampMonotonic": "15234031", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "1844674407370 2021-03-09T08:52:11.909 INFO:teuthology.task.ansible.out:9551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 08:49:09 UTC", "InactiveExitTimestampMonotonic": "14704295", "InvocationID": "5f009e2a70584832a8b8a6fa5755103d", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127781", "LimitNPROCSoft": "127781", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127781", "LimitSIGPENDINGSoft": "127781", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1370", "MemoryAccounting": "yes", "MemoryCurrent": "10141696", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPol 2021-03-09T08:52:11.911 INFO:teuthology.task.ansible.out:icy": "n/a", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:49:09 UTC", "StateChangeTimestampMonotonic": "15234054", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "38334", "TimeoutAbortUSec" 2021-03-09T08:52:11.912 INFO:teuthology.task.ansible.out:: "1min 30s", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-09T08:52:11.948 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:11 +0000 (0:00:00.185) 0:02:06.970 ********* 2021-03-09T08:52:12.004 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-09T08:52:12.007 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009485", "end": "2021-03-09 08:52:11.977820", "rc": 0, "start": "2021-03-09 08:52:11.968335", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:12.048 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:12 +0000 (0:00:00.099) 0:02:07.070 ********* 2021-03-09T08:52:12.069 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-09T08:52:12.071 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:12.097 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:12 +0000 (0:00:00.049) 0:02:07.119 ********* 2021-03-09T08:52:12.177 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi005.front.sepia.ceph.com] => (item={'block_used': 13728278, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226366830, 'size_available': 927198535680, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 113650, 'block_size': 4096, 'inode_available': 60941326}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226366830, "block_size": 4096, "block_total": 240095108, "block_used": 13728278, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941326, "inode_total": 61054976, "inode_used": 113650, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 927198535680, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:12.207 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:12 +0000 (0:00:00.108) 0:02:07.228 ********* 2021-03-09T08:52:12.225 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T08:52:12.227 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:12.252 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:12 +0000 (0:00:00.045) 0:02:07.274 ********* 2021-03-09T08:52:12.281 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-09T08:52:12.316 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:12 +0000 (0:00:00.063) 0:02:07.337 ********* 2021-03-09T08:52:12.338 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-09T08:52:12.340 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:12.365 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:12 +0000 (0:00:00.049) 0:02:07.387 ********* 2021-03-09T08:52:12.439 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T08:52:12.440 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:12.461 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-09T08:52:12.463 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:12.486 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:12 +0000 (0:00:00.120) 0:02:07.508 ********* 2021-03-09T08:52:12.568 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-09T08:52:12.569 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:12.593 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:12 +0000 (0:00:00.106) 0:02:07.615 ********* 2021-03-09T08:52:12.673 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:12.697 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:12 +0000 (0:00:00.104) 0:02:07.719 ********* 2021-03-09T08:52:12.788 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-09T08:52:12.790 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:12.825 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:12 +0000 (0:00:00.127) 0:02:07.846 ********* 2021-03-09T08:52:12.897 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:12.923 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:12 +0000 (0:00:00.098) 0:02:07.945 ********* 2021-03-09T08:52:13.025 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:13.053 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:13 +0000 (0:00:00.130) 0:02:08.075 ********* 2021-03-09T08:52:13.138 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:13.166 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:13 +0000 (0:00:00.113) 0:02:08.188 ********* 2021-03-09T08:52:13.236 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:13.238 INFO:teuthology.task.ansible.out:TASK [testnode : Install packages] ********************************************* skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:13.260 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:13 +0000 (0:00:00.094) 0:02:08.282 ********* 2021-03-09T08:52:13.345 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T08:52:13.347 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:13.375 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:13 +0000 (0:00:00.114) 0:02:08.397 ********* 2021-03-09T08:52:13.455 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-09T08:52:13.457 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:13.478 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:13 +0000 (0:00:00.103) 0:02:08.500 ********* 2021-03-09T08:52:13.552 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T08:52:13.554 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:13.575 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:13 +0000 (0:00:00.097) 0:02:08.597 ********* 2021-03-09T08:52:13.655 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T08:52:13.657 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:13.681 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:13 +0000 (0:00:00.105) 0:02:08.703 ********* 2021-03-09T08:52:13.753 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T08:52:13.755 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:13.777 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:13 +0000 (0:00:00.096) 0:02:08.799 ********* 2021-03-09T08:52:13.860 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T08:52:13.861 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:13.885 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:13 +0000 (0:00:00.107) 0:02:08.907 ********* 2021-03-09T08:52:13.967 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T08:52:13.969 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:13.993 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:13 +0000 (0:00:00.107) 0:02:09.015 ********* 2021-03-09T08:52:14.056 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T08:52:14.058 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxXOtc3AyqR7lPBG3vc0Tj7gVPY3hFIXzvW'], 'uuids': ['cad3b29d-7c31-42da-91ec-ae1306b95296']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.482482", "end": "2021-03-09 08:52:14.027002", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxXOtc3AyqR7lPBG3vc0Tj7gVPY3hFIXzvW"], "labels": [], "masters": [], "uuids": ["cad3b29d-7c31-42da-91ec-ae1306b95296"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:12.544520", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:52:14.069 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:14.080 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:14.082 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:14.106 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:14 +0000 (0:00:00.113) 0:02:09.128 ********* 2021-03-09T08:52:14.187 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T08:52:14.189 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:14.211 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:14 +0000 (0:00:00.104) 0:02:09.233 ********* 2021-03-09T08:52:14.282 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T08:52:14.283 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:14.307 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:14 +0000 (0:00:00.095) 0:02:09.328 ********* 2021-03-09T08:52:14.387 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T08:52:14.389 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:14.413 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:14 +0000 (0:00:00.106) 0:02:09.435 ********* 2021-03-09T08:52:14.493 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:14.516 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:14 +0000 (0:00:00.103) 0:02:09.538 ********* 2021-03-09T08:52:14.596 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T08:52:14.598 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:14.633 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:14 +0000 (0:00:00.116) 0:02:09.655 ********* 2021-03-09T08:52:14.635 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:14.705 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T08:52:14.706 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:14.729 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:14 +0000 (0:00:00.096) 0:02:09.751 ********* 2021-03-09T08:52:14.809 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-03-09T08:52:14.811 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:14.834 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:14 +0000 (0:00:00.104) 0:02:09.855 ********* 2021-03-09T08:52:15.018 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-03-09T08:52:15.020 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-03-09T08:52:15.045 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:15 +0000 (0:00:00.211) 0:02:10.067 ********* 2021-03-09T08:52:15.339 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T08:52:15.341 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxXqQJ1Bvq8aYzVldUh3sduMKHIljPSogno'], 'uuids': ['46432480-ad55-49d6-83d5-c3aa2cafb7eb']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.119173", "end": "2021-03-09 08:52:15.302046", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxXqQJ1Bvq8aYzVldUh3sduMKHIljPSogno"], "labels": [], "masters": [], "uuids": ["46432480-ad55-49d6-83d5-c3aa2cafb7eb"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:14.182873", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:52:15.341 INFO:teuthology.task.ansible.out:skipping: [smithi005.front 2021-03-09T08:52:15.343 INFO:teuthology.task.ansible.out:.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQ1L', 'wwn-0x5000c500871a93fb'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQ1L-part1', 'wwn-0x5000c500871a93fb-part1'], 'uuids': ['765c9454-acf9-4ab1-8c0e-444ef075a2c6']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871a93fb', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HQ1L', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQ1L", "wwn-0x5000c500871a93fb"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQ1L-part1", "wwn-0x5000c500871a93fb-part1"], "labels": [], "masters": [], "uuids": ["765c9454-acf9-4ab1-8c0e-444ef075a2c6"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HQ1L", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a93fb"}}, "skip_reason": "Condition 2021-03-09T08:52:15.345 INFO:teuthology.task.ansible.out:al result was False"} 2021-03-09T08:52:15.450 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T08:52:15.452 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-03-09T08:52:15.483 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:15 +0000 (0:00:00.437) 0:02:10.504 ********* 2021-03-09T08:52:15.565 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T08:52:15.568 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:15.590 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:15 +0000 (0:00:00.107) 0:02:10.612 ********* 2021-03-09T08:52:16.033 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-03-09T08:52:16.062 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:16 +0000 (0:00:00.471) 0:02:11.084 ********* 2021-03-09T08:52:16.505 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-03-09T08:52:16.507 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-03-09T08:52:16.529 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:16 +0000 (0:00:00.466) 0:02:11.550 ********* 2021-03-09T08:52:16.670 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T08:52:16.672 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-jGkBUF-qWoQ-YWDX-XiOr-4QrE-NWrM-m4rBq5', 'nvme-INTEL_SSDPED1D480GA_PHMB751300EW480DGN', 'nvme-nvme.8086-50484d42373531333030455734383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.186590", "end": "2021-03-09 08:52:16.633057", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-jGkBUF-qWoQ-YWDX-XiOr-4QrE-NWrM-m4rBq5", "nvme-INTEL_SSDPED1D480GA_PHMB751300EW480DGN", "nvme-nvme.8086-50484d42373531333030455734383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:15.446467", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nGPT data structures destroyed! You 2021-03-09T08:52:16.674 INFO:teuthology.task.ansible.out:may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:52:16.683 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:16.696 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:16.955 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-03-09T08:52:16.979 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:16 +0000 (0:00:00.450) 0:02:12.001 ********* 2021-03-09T08:52:17.178 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-03-09T08:52:17.179 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:52:17.211 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:17 +0000 (0:00:00.231) 0:02:12.233 ********* 2021-03-09T08:52:17.291 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-03-09T08:52:17.293 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:17.334 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:17 +0000 (0:00:00.122) 0:02:12.355 ********* 2021-03-09T08:52:17.408 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-03-09T08:52:17.410 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:17.432 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:17 +0000 (0:00:00.098) 0:02:12.454 ********* 2021-03-09T08:52:17.513 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-03-09T08:52:17.513 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:17.536 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:17 +0000 (0:00:00.104) 0:02:12.558 ********* 2021-03-09T08:52:17.618 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-03-09T08:52:17.620 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:17.648 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:17 +0000 (0:00:00.111) 0:02:12.670 ********* 2021-03-09T08:52:17.743 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-09T08:52:17.745 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:17.768 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:17 +0000 (0:00:00.120) 0:02:12.790 ********* 2021-03-09T08:52:17.860 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:52:17.862 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-09T08:52:17.889 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:17 +0000 (0:00:00.116) 0:02:12.906 ********* 2021-03-09T08:52:17.986 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T08:52:17.988 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxX0hXqh1r52OcdbFieUWZmrMdZ41dz6hGU'], 'uuids': ['b79d6f8e-a224-480a-aa57-ba2e4e22c670']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.147167", "end": "2021-03-09 08:52:17.953746", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxX0hXqh1r52OcdbFieUWZmrMdZ41dz6hGU"], "labels": [], "masters": [], "uuids": ["b79d6f8e-a224-480a-aa57-ba2e4e22c670"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:16.806579", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:52:18.000 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.000 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:18.002 INFO:teuthology.task.ansible.out:TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi146.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.003 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:18.009 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T08:52:18.010 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi146.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.021 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.032 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.043 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.055 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.066 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-03-09T08:52:18.067 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:18.078 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.080 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.090 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.092 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.102 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.104 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.115 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.117 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.152 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:18 +0000 (0:00:00.267) 0:02:13.174 ********* 2021-03-09T08:52:18.424 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-03-09T08:52:18.426 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.007071", "end": "2021-03-09 08:52:18.397966", "rc": 0, "start": "2021-03-09 08:52:18.390895", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock"]} 2021-03-09T08:52:18.452 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:18 +0000 (0:00:00.299) 0:02:13.474 ********* 2021-03-09T08:52:18.532 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:18.534 INFO:teuthology.task.ansible.out:TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.560 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:18 +0000 (0:00:00.108) 0:02:13.582 ********* 2021-03-09T08:52:18.650 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.678 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:18 +0000 (0:00:00.118) 0:02:13.700 ********* 2021-03-09T08:52:18.769 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-03-09T08:52:18.771 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:18.799 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:18 +0000 (0:00:00.116) 0:02:13.817 ********* 2021-03-09T08:52:19.284 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T08:52:19.286 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxXp2aP1QTdeCcCu5czGflxCZhCak1y9h3V'], 'uuids': ['12d81fe1-6902-40e5-97da-9270573951ee']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.135392", "end": "2021-03-09 08:52:19.247886", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxXp2aP1QTdeCcCu5czGflxCZhCak1y9h3V"], "labels": [], "masters": [], "uuids": ["12d81fe1-6902-40e5-97da-9270573951ee"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:18.112494", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:52:19.297 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:19.299 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:19.521 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-03-09T08:52:19.522 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1615279794, "cache_updated": false, "changed": false} 2021-03-09T08:52:19.551 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:19 +0000 (0:00:00.755) 0:02:14.573 ********* 2021-03-09T08:52:19.621 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-09T08:52:19.622 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:19.623 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:19.646 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:19 +0000 (0:00:00.094) 0:02:14.668 ********* 2021-03-09T08:52:19.738 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T08:52:19.740 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item={'block_used': 13728276, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226366832, 'size_available': 927198543872, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 113649, 'block_size': 4096, 'inode_available': 60941327}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226366832, "block_size": 4096, "block_total": 240095108, "block_used": 13728276, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941327, "inode_total": 61054976, "inode_used": 113649, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 927198543872, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}} 2021-03-09T08:52:19.763 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:19 +0000 (0:00:00.116) 0:02:14.784 ********* 2021-03-09T08:52:19.970 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-09T08:52:19.972 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009537", "end": "2021-03-09 08:52:20.376649", "rc": 0, "start": "2021-03-09 08:52:20.367112", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:19.996 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:19 +0000 (0:00:00.233) 0:02:15.018 ********* 2021-03-09T08:52:20.086 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-03-09T08:52:20.088 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'block_used': 13728276, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226366832, 'size_available': 927198543872, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 113649, 'block_size': 4096, 'inode_available': 60941327}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226366832, "block_size": 4096, "block_total": 240095108, "block_used": 13728276, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941327, "inode_total": 61054976, "inode_used": 113649, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 927198543872, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:20.111 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:20 +0000 (0:00:00.115) 0:02:15.133 ********* 2021-03-09T08:52:20.202 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-09T08:52:20.226 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:20 +0000 (0:00:00.114) 0:02:15.248 ********* 2021-03-09T08:52:20.330 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T08:52:20.332 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:20.553 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxXUS6yfUqGVg04eb4ORdkBZNgx10Tuk86j'], 'uuids': ['bcf43f4c-2814-4a5b-bc1f-071e93829bfb']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.126493", "end": "2021-03-09 08:52:20.524864", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxXUS6yfUqGVg04eb4ORdkBZNgx10Tuk86j"], "labels": [], "masters": [], "uuids": ["bcf43f4c-2814-4a5b-bc1f-071e93829bfb"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:19.398371", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:52:20.600 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:20 +0000 (0:00:00.373) 0:02:15.621 ********* 2021-03-09T08:52:20.691 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T08:52:20.693 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:20.934 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxXOtc3AyqR7lPBG3vc0Tj7gVPY3hFIXzvW'], 'uuids': ['cad3b29d-7c31-42da-91ec-ae1306b95296']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.115799", "end": "2021-03-09 08:52:20.906216", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxXOtc3AyqR7lPBG3vc0Tj7gVPY3hFIXzvW"], "labels": [], "masters": [], "uuids": ["cad3b29d-7c31-42da-91ec-ae1306b95296"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:20.790417", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.111354 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.111354 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:20.946 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:20.957 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:21.200 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxXqQJ1Bvq8aYzVldUh3sduMKHIljPSogno'], 'uuids': ['46432480-ad55-49d6-83d5-c3aa2cafb7eb']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.116325", "end": "2021-03-09 08:52:21.173237", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxXqQJ1Bvq8aYzVldUh3sduMKHIljPSogno"], "labels": [], "masters": [], "uuids": ["46432480-ad55-49d6-83d5-c3aa2cafb7eb"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:21.056912", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.111469 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.111469 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:21.213 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQ1L', 'wwn-0x5000c500871a93fb'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQ1L-part1', 'wwn-0x5000c500871a93fb-part1'], 'uuids': ['765c9454-acf9-4ab1-8c0e-444ef075a2c6']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871a93fb', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HQ1L', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQ1L", "wwn-0x5000c500871a93fb"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQ1L-part1", "wwn-0x5000c500871a93fb-part1"], "labels": [], "masters": [], "uuids": ["765c9454-acf9-4ab1-8c0e-444ef075a2c6"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HQ1L", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a93fb"}}, 2021-03-09T08:52:21.214 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2021-03-09T08:52:21.387 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-jGkBUF-qWoQ-YWDX-XiOr-4QrE-NWrM-m4rBq5', 'nvme-INTEL_SSDPED1D480GA_PHMB751300EW480DGN', 'nvme-nvme.8086-50484d42373531333030455734383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.054161", "end": "2021-03-09 08:52:21.357515", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-jGkBUF-qWoQ-YWDX-XiOr-4QrE-NWrM-m4rBq5", "nvme-INTEL_SSDPED1D480GA_PHMB751300EW480DGN", "nvme-nvme.8086-50484d42373531333030455734383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:21.303354", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0494081 s, 2.3 GB/s", "stderr_lines": [" 2021-03-09T08:52:21.389 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0494081 s, 2.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:21.399 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:21.400 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:21.642 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxX0hXqh1r52OcdbFieUWZmrMdZ41dz6hGU'], 'uuids': ['b79d6f8e-a224-480a-aa57-ba2e4e22c670']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.110904", "end": "2021-03-09 08:52:21.610526", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxX0hXqh1r52OcdbFieUWZmrMdZ41dz6hGU"], "labels": [], "masters": [], "uuids": ["b79d6f8e-a224-480a-aa57-ba2e4e22c670"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:21.499622", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.106148 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.106148 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:21.654 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:21.655 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:21.900 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxXp2aP1QTdeCcCu5czGflxCZhCak1y9h3V'], 'uuids': ['12d81fe1-6902-40e5-97da-9270573951ee']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.117658", "end": "2021-03-09 08:52:21.873942", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxXp2aP1QTdeCcCu5czGflxCZhCak1y9h3V"], "labels": [], "masters": [], "uuids": ["12d81fe1-6902-40e5-97da-9270573951ee"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:21.756284", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.112848 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.112848 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:21.914 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:21.936 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T08:52:21.938 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1mQ7wyZZOzla87kA6uq5nIb6fcBb02pDOE'], 'uuids': ['af816d62-a9b3-43bc-b510-945a2828fd70']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.477109", "end": "2021-03-09 08:52:22.327904", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1mQ7wyZZOzla87kA6uq5nIb6fcBb02pDOE"], "labels": [], "masters": [], "uuids": ["af816d62-a9b3-43bc-b510-945a2828fd70"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:20.850795", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi146.front 2021-03-09T08:52:21.940 INFO:teuthology.task.ansible.out:.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:21.950 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:22.163 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T08:52:22.165 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxXUS6yfUqGVg04eb4ORdkBZNgx10Tuk86j'], 'uuids': ['bcf43f4c-2814-4a5b-bc1f-071e93829bfb']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.115014", "end": "2021-03-09 08:52:22.135938", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-lrh4Ff1LYi87lf1RUiGy3OT4BR7i7uxXUS6yfUqGVg04eb4ORdkBZNgx10Tuk86j"], "labels": [], "masters": [], "uuids": ["bcf43f4c-2814-4a5b-bc1f-071e93829bfb"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:22.020924", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.110139 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.110139 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:22.198 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:22 +0000 (0:00:01.598) 0:02:17.220 ********* 2021-03-09T08:52:22.555 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-09T08:52:22.557 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.136233", "end": "2021-03-09 08:52:22.457116", "rc": 0, "start": "2021-03-09 08:52:22.320883", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:22.581 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:22 +0000 (0:00:00.383) 0:02:17.603 ********* 2021-03-09T08:52:22.809 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-03-09T08:52:22.811 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.080834", "end": "2021-03-09 08:52:22.777745", "rc": 0, "start": "2021-03-09 08:52:22.696911", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:22.844 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:22 +0000 (0:00:00.262) 0:02:17.866 ********* 2021-03-09T08:52:22.914 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-09T08:52:22.992 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:22 +0000 (0:00:00.147) 0:02:18.014 ********* 2021-03-09T08:52:23.020 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-03-09T08:52:23.022 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:23.046 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:23 +0000 (0:00:00.053) 0:02:18.068 ********* 2021-03-09T08:52:23.074 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-03-09T08:52:23.076 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:23.097 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:23 +0000 (0:00:00.051) 0:02:18.119 ********* 2021-03-09T08:52:23.125 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T08:52:23.127 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:23.161 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:23 +0000 (0:00:00.063) 0:02:18.183 ********* 2021-03-09T08:52:23.193 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-09T08:52:23.195 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:23.217 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:23 +0000 (0:00:00.055) 0:02:18.239 ********* 2021-03-09T08:52:23.227 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:23.229 INFO:teuthology.task.ansible.out:TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T08:52:23.230 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1m6WYqFovaM6N4593ixCcpatXnmEKIDYcJ'], 'uuids': ['1d030fae-51cc-4b28-bf76-652d00fce836']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:00.708819", "end": "2021-03-09 08:52:23.178969", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1m6WYqFovaM6N4593ixCcpatXnmEKIDYcJ"], "labels": [], "masters": [], "uuids": ["1d030fae-51cc-4b28-bf76-652d00fce836"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:22.470150", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now pa 2021-03-09T08:52:23.231 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or", "other utilities."]} skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFPH', 'wwn-0x5000c50092925063'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFPH-part1', 'wwn-0x5000c50092925063-part1'], 'uuids': ['765c9454-acf9-4ab1-8c0e-444ef075a2c6']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092925063', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XFPH', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFPH", "wwn-0x5000c50092925063"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFPH-part1", "wwn-0x5000c50092925063-part1"], "labels": [], "masters": [], "uuids": ["765c9454-acf9-4ab1-8c0e-444ef075a2c6"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFPH", "size": "931.51 GB", "support_discard": "0", "vend 2021-03-09T08:52:23.233 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c50092925063"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:23.359 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-09T08:52:23.361 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:52:23.384 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:23 +0000 (0:00:00.166) 0:02:18.406 ********* 2021-03-09T08:52:23.433 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T08:52:23.437 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'block_used': 13728278, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226366830, 'size_available': 927198535680, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 113650, 'block_size': 4096, 'inode_available': 60941326}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226366830, "block_size": 4096, "block_total": 240095108, "block_used": 13728278, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941326, "inode_total": 61054976, "inode_used": 113650, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 927198535680, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:23.458 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:23 +0000 (0:00:00.073) 0:02:18.480 ********* 2021-03-09T08:52:23.490 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-09T08:52:23.492 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:23.515 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:23 +0000 (0:00:00.056) 0:02:18.537 ********* 2021-03-09T08:52:23.543 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:52:23.545 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:23.567 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:23 +0000 (0:00:00.052) 0:02:18.589 ********* 2021-03-09T08:52:23.596 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:52:23.598 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:23.620 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:23 +0000 (0:00:00.053) 0:02:18.642 ********* 2021-03-09T08:52:23.648 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-09T08:52:23.650 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:23.674 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:23 +0000 (0:00:00.053) 0:02:18.696 ********* 2021-03-09T08:52:23.702 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-09T08:52:23.703 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:23.727 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:23 +0000 (0:00:00.052) 0:02:18.748 ********* 2021-03-09T08:52:23.754 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-09T08:52:23.755 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:23.777 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:23 +0000 (0:00:00.050) 0:02:18.799 ********* 2021-03-09T08:52:24.530 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T08:52:24.532 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-QOzh2p-FEgG-Bq5B-56t7-5t2L-iiD6-d5eb6e', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300GB400BGN', 'nvme-nvme.8086-43564654363233333030474234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.182941", "end": "2021-03-09 08:52:24.501396", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-QOzh2p-FEgG-Bq5B-56t7-5t2L-iiD6-d5eb6e", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GB400BGN", "nvme-nvme.8086-43564654363233333030474234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:23.318455", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nGPT data structures d 2021-03-09T08:52:24.533 INFO:teuthology.task.ansible.out:estroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:52:24.542 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:24.543 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:24.630 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-03-09T08:52:24.631 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-09T08:52:24.657 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:24 +0000 (0:00:00.879) 0:02:19.679 ********* 2021-03-09T08:52:25.409 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-09T08:52:25.411 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:52:25.811 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:52:25.822 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T08:52:25.825 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1mGU4tb7GFPA6xBJlmh7VygE8hiyTM5Yko'], 'uuids': ['983fd260-a387-4b72-ad39-1abf4e90ee56']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.144570", "end": "2021-03-09 08:52:25.790708", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1mGU4tb7GFPA6xBJlmh7VygE8hiyTM5Yko"], "labels": [], "masters": [], "uuids": ["983fd260-a387-4b72-ad39-1abf4e90ee56"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:24.646138", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:52:25.835 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:25.837 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:26.241 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-09T08:52:26.244 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:52:26.635 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:52:27.003 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:52:27.029 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:27 +0000 (0:00:02.371) 0:02:22.051 ********* 2021-03-09T08:52:27.102 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T08:52:27.104 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1mq9cj2c1pj3X0xVoED6RS0q6JMVv24iEl'], 'uuids': ['061e43c2-3670-4aff-97d6-a50564a9112a']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.124731", "end": "2021-03-09 08:52:27.065985", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1mq9cj2c1pj3X0xVoED6RS0q6JMVv24iEl"], "labels": [], "masters": [], "uuids": ["061e43c2-3670-4aff-97d6-a50564a9112a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:25.941254", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:52:27.115 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:27.200 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T08:52:27.202 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-09T08:52:27.229 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:27 +0000 (0:00:00.199) 0:02:22.250 ********* 2021-03-09T08:52:27.387 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-09T08:52:27.389 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:52:27.518 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:52:27.645 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:52:27.773 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:52:27.786 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:27.811 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:27 +0000 (0:00:00.582) 0:02:22.833 ********* 2021-03-09T08:52:27.842 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-03-09T08:52:27.844 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:27.870 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:27 +0000 (0:00:00.058) 0:02:22.892 ********* 2021-03-09T08:52:27.899 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-03-09T08:52:27.900 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:27.924 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:27 +0000 (0:00:00.053) 0:02:22.946 ********* 2021-03-09T08:52:28.069 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-09T08:52:28.071 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:52:28.093 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:28 +0000 (0:00:00.169) 0:02:23.115 ********* 2021-03-09T08:52:28.143 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-09T08:52:28.168 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:28 +0000 (0:00:00.074) 0:02:23.190 ********* 2021-03-09T08:52:28.391 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T08:52:28.393 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1msBrZVOh83u4LhBaCz1edfYTc6hfwiCLI'], 'uuids': ['60898493-3c66-48eb-8ff4-96a5d49cfac5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.144699", "end": "2021-03-09 08:52:28.354758", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1msBrZVOh83u4LhBaCz1edfYTc6hfwiCLI"], "labels": [], "masters": [], "uuids": ["60898493-3c66-48eb-8ff4-96a5d49cfac5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:27.210059", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:52:28.434 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:28 +0000 (0:00:00.266) 0:02:23.456 ********* 2021-03-09T08:52:28.506 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T08:52:28.508 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:28.688 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi005.front.sepia.ceph.com] => {"changed": true} 2021-03-09T08:52:28.715 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:28 +0000 (0:00:00.280) 0:02:23.737 ********* 2021-03-09T08:52:28.830 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T08:52:28.832 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1mQ7wyZZOzla87kA6uq5nIb6fcBb02pDOE'], 'uuids': ['af816d62-a9b3-43bc-b510-945a2828fd70']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.163837", "end": "2021-03-09 08:52:28.793012", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1mQ7wyZZOzla87kA6uq5nIb6fcBb02pDOE"], "labels": [], "masters": [], "uuids": ["af816d62-a9b3-43bc-b510-945a2828fd70"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:28.629175", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.159892 s, 721 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.159892 s, 721 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:28.832 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 2021-03-09T08:52:28.839 INFO:teuthology.task.ansible.out:'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:28.855 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:29.149 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1m6WYqFovaM6N4593ixCcpatXnmEKIDYcJ'], 'uuids': ['1d030fae-51cc-4b28-bf76-652d00fce836']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.166535", "end": "2021-03-09 08:52:29.118003", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1m6WYqFovaM6N4593ixCcpatXnmEKIDYcJ"], "labels": [], "masters": [], "uuids": ["1d030fae-51cc-4b28-bf76-652d00fce836"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:28.951468", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.161737 s, 713 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.161737 s, 713 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:29.162 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFPH', 'wwn-0x5000c50092925063'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFPH-part1', 'wwn-0x5000c50092925063-part1'], 'uuids': ['765c9454-acf9-4ab1-8c0e-444ef075a2c6']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092925063', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XFPH', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFPH", "wwn-0x5000c50092925063"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFPH-part1", "wwn-0x5000c50092925063-part1"], "labels": [], "masters": [], "uuids": ["765c9454-acf9-4ab1-8c0e-444ef075a2c6"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFPH", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092925063"}}, 2021-03-09T08:52:29.163 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2021-03-09T08:52:29.249 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-03-09T08:52:29.251 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-09T08:52:29.281 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:29 +0000 (0:00:00.566) 0:02:24.303 ********* 2021-03-09T08:52:29.311 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-03-09T08:52:29.313 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:29.355 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:29 +0000 (0:00:00.073) 0:02:24.377 ********* 2021-03-09T08:52:29.367 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T08:52:29.370 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-QOzh2p-FEgG-Bq5B-56t7-5t2L-iiD6-d5eb6e', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300GB400BGN', 'nvme-nvme.8086-43564654363233333030474234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.054351", "end": "2021-03-09 08:52:29.313611", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-QOzh2p-FEgG-Bq5B-56t7-5t2L-iiD6-d5eb6e", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GB400BGN", "nvme-nvme.8086-43564654363233333030474234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:29.259260", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0496519 s, 2.3 GB/s", "std 2021-03-09T08:52:29.372 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0496519 s, 2.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:29.373 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:29.375 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:29.397 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-03-09T08:52:29.399 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:29.433 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:29 +0000 (0:00:00.077) 0:02:24.455 ********* 2021-03-09T08:52:29.461 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-09T08:52:29.464 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:29.499 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:29 +0000 (0:00:00.066) 0:02:24.521 ********* 2021-03-09T08:52:29.528 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-09T08:52:29.530 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:29.570 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:29 +0000 (0:00:00.070) 0:02:24.591 ********* 2021-03-09T08:52:29.706 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T08:52:29.708 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1mGU4tb7GFPA6xBJlmh7VygE8hiyTM5Yko'], 'uuids': ['983fd260-a387-4b72-ad39-1abf4e90ee56']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.165705", "end": "2021-03-09 08:52:29.669991", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1mGU4tb7GFPA6xBJlmh7VygE8hiyTM5Yko"], "labels": [], "masters": [], "uuids": ["983fd260-a387-4b72-ad39-1abf4e90ee56"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:29.504286", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.161996 s, 712 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.161996 s, 712 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:29.710 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 's 2021-03-09T08:52:29.712 INFO:teuthology.task.ansible.out:ize': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:29.723 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:29.769 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T08:52:29.796 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:29 +0000 (0:00:00.226) 0:02:24.817 ********* 2021-03-09T08:52:29.977 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-03-09T08:52:29.979 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:52:30.009 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:30 +0000 (0:00:00.213) 0:02:25.031 ********* 2021-03-09T08:52:30.036 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T08:52:30.037 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1mq9cj2c1pj3X0xVoED6RS0q6JMVv24iEl'], 'uuids': ['061e43c2-3670-4aff-97d6-a50564a9112a']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.164574", "end": "2021-03-09 08:52:29.992507", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1mq9cj2c1pj3X0xVoED6RS0q6JMVv24iEl"], "labels": [], "masters": [], "uuids": ["061e43c2-3670-4aff-97d6-a50564a9112a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:29.827933", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.159626 s, 723 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.159626 s, 723 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:30.039 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:30.334 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1msBrZVOh83u4LhBaCz1edfYTc6hfwiCLI'], 'uuids': ['60898493-3c66-48eb-8ff4-96a5d49cfac5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.159061", "end": "2021-03-09 08:52:30.298261", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-dBRT3J5enOvbRVDyWc3t81sbWOsIEG1msBrZVOh83u4LhBaCz1edfYTc6hfwiCLI"], "labels": [], "masters": [], "uuids": ["60898493-3c66-48eb-8ff4-96a5d49cfac5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:52:30.139200", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154256 s, 748 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154256 s, 748 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:30.376 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:30 +0000 (0:00:00.366) 0:02:25.398 ********* 2021-03-09T08:52:30.456 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-03-09T08:52:30.459 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-03-09T08:52:30.497 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:30 +0000 (0:00:00.120) 0:02:25.519 ********* 2021-03-09T08:52:30.687 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-09T08:52:30.690 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.147181", "end": "2021-03-09 08:52:30.657612", "rc": 0, "start": "2021-03-09 08:52:30.510431", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:30.713 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:30 +0000 (0:00:00.216) 0:02:25.735 ********* 2021-03-09T08:52:30.901 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-03-09T08:52:30.903 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-03-09T08:52:30.934 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:30 +0000 (0:00:00.220) 0:02:25.956 ********* 2021-03-09T08:52:30.952 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-03-09T08:52:30.954 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.094606", "end": "2021-03-09 08:52:30.926147", "rc": 0, "start": "2021-03-09 08:52:30.831541", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:30.979 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:30 +0000 (0:00:00.045) 0:02:26.001 ********* 2021-03-09T08:52:31.008 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-09T08:52:31.038 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:31 +0000 (0:00:00.058) 0:02:26.060 ********* 2021-03-09T08:52:31.066 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-03-09T08:52:31.067 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:31.095 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:31 +0000 (0:00:00.054) 0:02:26.114 ********* 2021-03-09T08:52:31.125 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-03-09T08:52:31.127 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:31.150 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:31 +0000 (0:00:00.057) 0:02:26.172 ********* 2021-03-09T08:52:31.179 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T08:52:31.181 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:31.206 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:31 +0000 (0:00:00.055) 0:02:26.228 ********* 2021-03-09T08:52:31.235 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-09T08:52:31.236 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:31.259 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:31 +0000 (0:00:00.052) 0:02:26.280 ********* 2021-03-09T08:52:31.395 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-09T08:52:31.398 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:52:31.420 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:31 +0000 (0:00:00.161) 0:02:26.442 ********* 2021-03-09T08:52:31.470 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T08:52:31.472 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'block_used': 13728276, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226366832, 'size_available': 927198543872, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 113649, 'block_size': 4096, 'inode_available': 60941327}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226366832, "block_size": 4096, "block_total": 240095108, "block_used": 13728276, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941327, "inode_total": 61054976, "inode_used": 113649, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 927198543872, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:31.497 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:31 +0000 (0:00:00.076) 0:02:26.518 ********* 2021-03-09T08:52:31.536 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-09T08:52:31.538 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:31.565 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:31 +0000 (0:00:00.068) 0:02:26.587 ********* 2021-03-09T08:52:31.594 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-03-09T08:52:31.596 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"cache_update_time": 1615279787, "cache_updated": false, "changed": false} 2021-03-09T08:52:31.723 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:31 +0000 (0:00:00.158) 0:02:26.745 ********* 2021-03-09T08:52:31.732 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:52:31.733 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:31.759 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:31 +0000 (0:00:00.035) 0:02:26.780 ********* 2021-03-09T08:52:31.796 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:52:31.797 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:31.820 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:31 +0000 (0:00:00.061) 0:02:26.842 ********* 2021-03-09T08:52:31.858 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-09T08:52:31.860 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:31.883 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:31 +0000 (0:00:00.063) 0:02:26.905 ********* 2021-03-09T08:52:31.910 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-09T08:52:31.912 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:31.936 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:31 +0000 (0:00:00.051) 0:02:26.957 ********* 2021-03-09T08:52:31.975 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-09T08:52:31.976 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:31.999 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:31 +0000 (0:00:00.063) 0:02:27.021 ********* 2021-03-09T08:52:32.006 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-03-09T08:52:32.007 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.122483", "end": "2021-03-09 08:52:31.968857", "rc": 0, "start": "2021-03-09 08:52:31.846374", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.30.0/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.30.0/Amazon/S3.pm"]} 2021-03-09T08:52:32.033 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:32 +0000 (0:00:00.034) 0:02:27.055 ********* 2021-03-09T08:52:32.072 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-03-09T08:52:32.074 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:32.097 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:32 +0000 (0:00:00.063) 0:02:27.118 ********* 2021-03-09T08:52:32.124 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-03-09T08:52:32.126 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:32.148 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:32 +0000 (0:00:00.051) 0:02:27.170 ********* 2021-03-09T08:52:32.584 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-03-09T08:52:32.586 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-09T08:52:32.612 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:32 +0000 (0:00:00.463) 0:02:27.634 ********* 2021-03-09T08:52:32.823 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:32.825 INFO:teuthology.task.ansible.out:TASK [testnode : Install ntp package on apt based systems.] ******************** ok: [smithi005.front.sepia.ceph.com] => {"cache_update_time": 1615279787, "cache_updated": false, "changed": false} 2021-03-09T08:52:32.857 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:32 +0000 (0:00:00.244) 0:02:27.879 ********* 2021-03-09T08:52:32.886 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-03-09T08:52:32.888 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:32.927 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:32 +0000 (0:00:00.070) 0:02:27.949 ********* 2021-03-09T08:52:32.929 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:33.069 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-09T08:52:33.071 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:52:33.350 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-03-09T08:52:33.352 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-03-09T08:52:33.387 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:33 +0000 (0:00:00.460) 0:02:28.409 ********* 2021-03-09T08:52:33.450 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:33.479 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:33 +0000 (0:00:00.091) 0:02:28.501 ********* 2021-03-09T08:52:33.494 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-09T08:52:33.496 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:52:33.723 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-09T08:52:33.725 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 08:48:19 UTC", "ActiveEnterTimestampMonotonic": "13611192", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket basic.target sysinit.target -.mount systemd-tmpfiles-setup.service system.slice network.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 08:48:19 UTC", "AssertTimestampMonotonic": "13404518", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 08:48:19 UTC", "ConditionTimestampMonotonic": "13404516", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": 2021-03-09T08:52:33.727 INFO:teuthology.task.ansible.out: "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documentation": "man:ntpd(8)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "705", "ExecMainStartTimestamp": "Tue 2021-03-09 08:48:19 UTC", "ExecMainStartTimestampMonotonic": "13611179", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 08:48:19 UTC", "InactiveExitTimestampMonotonic": "13405762", "InvocationID": "9e5f3609b501424fb00fd29a67a99ab7", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "Lim 2021-03-09T08:52:33.728 INFO:teuthology.task.ansible.out:itCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127805", "LimitNPROCSoft": "127805", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127805", "LimitSIGPENDINGSoft": "127805", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "705", "MemoryAccounting": "yes", "MemoryCurrent": "4079616", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "Requi 2021-03-09T08:52:33.730 INFO:teuthology.task.ansible.out:resMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:48:19 UTC", "StateChangeTimestampMonotonic": "13611192", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "38341", "TimeoutAbortUSec": "1min 30s", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-09T08:52:33.754 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:33 +0000 (0:00:00.275) 0:02:28.776 ********* 2021-03-09T08:52:33.785 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:52:33.787 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-09T08:52:33.809 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:33 +0000 (0:00:00.055) 0:02:28.831 ********* 2021-03-09T08:52:33.860 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:33.862 INFO:teuthology.task.ansible.out:TASK [testnode : set_fact] ***************************************************** ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-03-09T08:52:33.885 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:33 +0000 (0:00:00.075) 0:02:28.907 ********* 2021-03-09T08:52:33.914 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-09T08:52:33.916 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:33.951 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:33 +0000 (0:00:00.066) 0:02:28.973 ********* 2021-03-09T08:52:34.303 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-09T08:52:34.305 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:52:34.716 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:52:34.755 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:34 +0000 (0:00:00.803) 0:02:29.777 ********* 2021-03-09T08:52:34.901 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T08:52:34.904 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-09T08:52:34.928 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:34 +0000 (0:00:00.173) 0:02:29.950 ********* 2021-03-09T08:52:34.948 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-03-09T08:52:34.950 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"cache_update_time": 1615279787, "cache_updated": false, "changed": false} 2021-03-09T08:52:34.980 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:34 +0000 (0:00:00.051) 0:02:30.001 ********* 2021-03-09T08:52:35.007 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-03-09T08:52:35.009 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:35.032 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:35 +0000 (0:00:00.052) 0:02:30.054 ********* 2021-03-09T08:52:35.034 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:35.098 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-09T08:52:35.101 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:52:35.216 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-03-09T08:52:35.218 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T08:52:35.250 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:35 +0000 (0:00:00.216) 0:02:30.271 ********* 2021-03-09T08:52:35.262 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-09T08:52:35.264 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:52:35.341 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:52:35.465 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:52:35.478 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:35.507 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:35 +0000 (0:00:00.257) 0:02:30.528 ********* 2021-03-09T08:52:35.536 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-03-09T08:52:35.539 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:35.564 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:35 +0000 (0:00:00.057) 0:02:30.586 ********* 2021-03-09T08:52:35.604 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:35.606 INFO:teuthology.task.ansible.out:TASK [testnode : Mount filesystems] ******************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:35.629 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:35 +0000 (0:00:00.064) 0:02:30.650 ********* 2021-03-09T08:52:35.742 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-03-09T08:52:35.744 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-03-09T08:52:35.775 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:35 +0000 (0:00:00.146) 0:02:30.797 ********* 2021-03-09T08:52:35.785 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-09T08:52:35.788 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:52:35.813 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:35 +0000 (0:00:00.037) 0:02:30.834 ********* 2021-03-09T08:52:35.864 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-09T08:52:35.866 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:35.893 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:35 +0000 (0:00:00.080) 0:02:30.915 ********* 2021-03-09T08:52:36.169 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:36.171 INFO:teuthology.task.ansible.out:TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi146.front.sepia.ceph.com] => {"changed": true} 2021-03-09T08:52:36.207 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:36 +0000 (0:00:00.314) 0:02:31.229 ********* 2021-03-09T08:52:36.453 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-03-09T08:52:36.455 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-09T08:52:36.479 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:36 +0000 (0:00:00.271) 0:02:31.501 ********* 2021-03-09T08:52:36.510 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-03-09T08:52:36.512 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:36.539 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:36 +0000 (0:00:00.059) 0:02:31.561 ********* 2021-03-09T08:52:36.579 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-03-09T08:52:36.581 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:36.608 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:36 +0000 (0:00:00.068) 0:02:31.630 ********* 2021-03-09T08:52:36.637 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-09T08:52:36.639 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:36.671 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:36 +0000 (0:00:00.063) 0:02:31.693 ********* 2021-03-09T08:52:36.673 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:36.700 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-09T08:52:36.702 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:36.729 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:36 +0000 (0:00:00.058) 0:02:31.751 ********* 2021-03-09T08:52:36.894 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-03-09T08:52:36.895 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T08:52:36.931 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:36 +0000 (0:00:00.201) 0:02:31.952 ********* 2021-03-09T08:52:37.088 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-03-09T08:52:37.090 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:52:37.114 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:37 +0000 (0:00:00.183) 0:02:32.136 ********* 2021-03-09T08:52:37.504 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-03-09T08:52:37.505 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-03-09T08:52:37.527 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:37 +0000 (0:00:00.413) 0:02:32.549 ********* 2021-03-09T08:52:37.897 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-03-09T08:52:37.900 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-03-09T08:52:37.929 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:37 +0000 (0:00:00.401) 0:02:32.951 ********* 2021-03-09T08:52:37.962 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-03-09T08:52:37.963 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.8/dist-packages (1.2.0)\nRequirement already satisfied: execnet in /usr/local/lib/python3.8/dist-packages (from remoto>=0.0.35) (1.8.0)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.8/dist-packages (from execnet->remoto>=0.0.35) (1.5)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.8/dist-packages (1.2.0)", "Requirement already satisfied: execnet in /usr/local/lib/python3.8/dist-packages (from remoto>=0.0.35) (1.8.0)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.8/dist-packages (from execnet->remoto>=0.0.35) (1.5)"], "version": null, "virtualenv": null} 2021-03-09T08:52:37.996 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:37 +0000 (0:00:00.067) 0:02:33.018 ********* 2021-03-09T08:52:38.543 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-03-09T08:52:38.545 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false} 2021-03-09T08:52:38.570 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:38 +0000 (0:00:00.574) 0:02:33.592 ********* 2021-03-09T08:52:38.600 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-03-09T08:52:38.602 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-03-09T08:52:38.630 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:38 +0000 (0:00:00.059) 0:02:33.651 ********* 2021-03-09T08:52:38.638 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-03-09T08:52:38.639 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1615279794, "cache_updated": false, "changed": false} 2021-03-09T08:52:38.669 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:38 +0000 (0:00:00.039) 0:02:33.691 ********* 2021-03-09T08:52:38.815 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-03-09T08:52:38.816 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1615279715.4960828, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1615279715.4920828, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 105181, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1615279715.4920828, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "1956422440", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-03-09T08:52:38.842 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:38 +0000 (0:00:00.173) 0:02:33.864 ********* 2021-03-09T08:52:38.949 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-03-09T08:52:38.951 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.118639", "end": "2021-03-09 08:52:38.916400", "rc": 0, "start": "2021-03-09 08:52:38.797761", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.30.0/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.30.0/Amazon/S3.pm"]} 2021-03-09T08:52:38.981 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:38 +0000 (0:00:00.138) 0:02:34.003 ********* 2021-03-09T08:52:39.034 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:39.059 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:39 +0000 (0:00:00.077) 0:02:34.081 ********* 2021-03-09T08:52:39.086 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-03-09T08:52:39.088 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:39.115 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:39 +0000 (0:00:00.056) 0:02:34.137 ********* 2021-03-09T08:52:39.253 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-03-09T08:52:39.255 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-03-09T08:52:39.280 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:39 +0000 (0:00:00.164) 0:02:34.301 ********* 2021-03-09T08:52:39.310 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-03-09T08:52:39.311 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-09T08:52:39.335 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:39 +0000 (0:00:00.055) 0:02:34.357 ********* 2021-03-09T08:52:39.363 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:39.389 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:39 +0000 (0:00:00.054) 0:02:34.411 ********* 2021-03-09T08:52:39.588 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-03-09T08:52:39.590 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-09T08:52:39.612 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:39 +0000 (0:00:00.223) 0:02:34.634 ********* 2021-03-09T08:52:39.769 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1615279794, "cache_updated": false, "changed": false} 2021-03-09T08:52:39.797 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:39 +0000 (0:00:00.184) 0:02:34.819 ********* 2021-03-09T08:52:39.816 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-03-09T08:52:39.817 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-09T08:52:39.843 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:39 +0000 (0:00:00.045) 0:02:34.864 ********* 2021-03-09T08:52:39.850 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-03-09T08:52:39.852 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:39.881 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:39 +0000 (0:00:00.038) 0:02:34.903 ********* 2021-03-09T08:52:39.890 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-03-09T08:52:39.891 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:39.918 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:39 +0000 (0:00:00.036) 0:02:34.939 ********* 2021-03-09T08:52:39.959 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-03-09T08:52:39.961 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi005.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:39.971 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:39.972 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:39.982 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:40.009 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:40 +0000 (0:00:00.091) 0:02:35.031 ********* 2021-03-09T08:52:40.050 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-03-09T08:52:40.051 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:40.062 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:40.063 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:40.073 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:40.075 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:40.102 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:40 +0000 (0:00:00.092) 0:02:35.124 ********* 2021-03-09T08:52:40.131 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:52:40.132 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:40.164 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:40 +0000 (0:00:00.061) 0:02:35.185 ********* 2021-03-09T08:52:40.192 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-03-09T08:52:40.194 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:40.218 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:40 +0000 (0:00:00.054) 0:02:35.240 ********* 2021-03-09T08:52:40.247 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-03-09T08:52:40.249 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-03-09T08:52:40.278 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:40 +0000 (0:00:00.058) 0:02:35.299 ********* 2021-03-09T08:52:40.286 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-03-09T08:52:40.287 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:40.322 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:40 +0000 (0:00:00.045) 0:02:35.344 ********* 2021-03-09T08:52:40.331 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-03-09T08:52:40.333 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:40.360 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:40 +0000 (0:00:00.037) 0:02:35.382 ********* 2021-03-09T08:52:40.368 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-03-09T08:52:40.369 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:40.395 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:40 +0000 (0:00:00.035) 0:02:35.417 ********* 2021-03-09T08:52:40.423 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-03-09T08:52:40.425 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:40.450 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:40 +0000 (0:00:00.054) 0:02:35.472 ********* 2021-03-09T08:52:40.489 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-09T08:52:40.491 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:40.518 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:40 +0000 (0:00:00.067) 0:02:35.540 ********* 2021-03-09T08:52:40.558 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-03-09T08:52:40.559 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:40.596 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:40 +0000 (0:00:00.078) 0:02:35.618 ********* 2021-03-09T08:52:40.613 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-09T08:52:40.616 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 08:49:07 UTC", "ActiveEnterTimestampMonotonic": "13651787", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice network.target sysinit.target -.mount systemd-tmpfiles-setup.service systemd-journald.socket basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 08:49:07 UTC", "AssertTimestampMonotonic": "13395218", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 08:49:07 UTC", "ConditionTimestampMonotonic": "13395218", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": 2021-03-09T08:52:40.618 INFO:teuthology.task.ansible.out: "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documentation": "man:ntpd(8)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "856", "ExecMainStartTimestamp": "Tue 2021-03-09 08:49:07 UTC", "ExecMainStartTimestampMonotonic": "13651773", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 08:49:07 UTC", "InactiveExitTimestampMonotonic": "13396188", "InvocationID": "1fad7e8040244710923bdccb8ca7763b", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "Lim 2021-03-09T08:52:40.619 INFO:teuthology.task.ansible.out:itCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127781", "LimitNPROCSoft": "127781", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127781", "LimitSIGPENDINGSoft": "127781", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "856", "MemoryAccounting": "yes", "MemoryCurrent": "3723264", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "Requi 2021-03-09T08:52:40.621 INFO:teuthology.task.ansible.out:resMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:49:07 UTC", "StateChangeTimestampMonotonic": "13651787", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "38334", "TimeoutAbortUSec": "1min 30s", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-09T08:52:40.662 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:40 +0000 (0:00:00.065) 0:02:35.683 ********* 2021-03-09T08:52:40.675 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-09T08:52:40.677 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:40.704 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:40 +0000 (0:00:00.042) 0:02:35.726 ********* 2021-03-09T08:52:40.712 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:40.714 INFO:teuthology.task.ansible.out:TASK [testnode : set_fact] ***************************************************** ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-09T08:52:40.747 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:40 +0000 (0:00:00.043) 0:02:35.769 ********* 2021-03-09T08:52:40.785 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:52:40.788 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-03-09T08:52:40.810 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:40 +0000 (0:00:00.062) 0:02:35.832 ********* 2021-03-09T08:52:40.840 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:40.864 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:40 +0000 (0:00:00.054) 0:02:35.886 ********* 2021-03-09T08:52:40.917 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-09T08:52:40.919 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011107", "end": "2021-03-09 08:52:40.881725", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-09 08:52:40.870618", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:41.085 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006154", "end": "2021-03-09 08:52:41.053867", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-09 08:52:41.047713", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:41.246 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005590", "end": "2021-03-09 08:52:41.219931", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-09 08:52:41.214341", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:41.473 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.027486", "end": "2021-03-09 08:52:41.436241", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-09 08:52:41.408755", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:41.518 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:41 +0000 (0:00:00.635) 0:02:36.522 ********* 2021-03-09T08:52:41.563 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-09T08:52:41.590 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:41 +0000 (0:00:00.090) 0:02:36.612 ********* 2021-03-09T08:52:41.797 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-03-09T08:52:41.799 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-03-09T08:52:41.843 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1615279794, "cache_updated": false, "changed": false} 2021-03-09T08:52:41.894 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:41 +0000 (0:00:00.300) 0:02:36.913 ********* 2021-03-09T08:52:41.967 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-03-09T08:52:41.970 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:42.005 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:42 +0000 (0:00:00.114) 0:02:37.027 ********* 2021-03-09T08:52:42.198 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T08:52:42.225 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:42 +0000 (0:00:00.220) 0:02:37.247 ********* 2021-03-09T08:52:42.685 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-03-09T08:52:42.687 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-03-09T08:52:42.710 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:42 +0000 (0:00:00.484) 0:02:37.731 ********* 2021-03-09T08:52:44.730 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-03-09T08:52:44.731 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.8/dist-packages (1.2.0)\nRequirement already satisfied: execnet in /usr/local/lib/python3.8/dist-packages (from remoto>=0.0.35) (1.8.0)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.8/dist-packages (from execnet->remoto>=0.0.35) (1.5)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.8/dist-packages (1.2.0)", "Requirement already satisfied: execnet in /usr/local/lib/python3.8/dist-packages (from remoto>=0.0.35) (1.8.0)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.8/dist-packages (from execnet->remoto>=0.0.35) (1.5)"], "version": null, "virtualenv": null} 2021-03-09T08:52:44.755 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:44 +0000 (0:00:02.044) 0:02:39.776 ********* 2021-03-09T08:52:45.272 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-03-09T08:52:45.274 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false} 2021-03-09T08:52:45.296 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:45 +0000 (0:00:00.541) 0:02:40.318 ********* 2021-03-09T08:52:45.334 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-03-09T08:52:45.336 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-03-09T08:52:45.359 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:45 +0000 (0:00:00.062) 0:02:40.381 ********* 2021-03-09T08:52:45.545 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-03-09T08:52:45.546 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1615279769.9078772, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1615279769.9038773, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 105181, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1615279769.9038773, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "1956422440", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-03-09T08:52:45.571 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:45 +0000 (0:00:00.211) 0:02:40.593 ********* 2021-03-09T08:52:45.746 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-03-09T08:52:45.748 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-03-09T08:52:45.774 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:45 +0000 (0:00:00.203) 0:02:40.796 ********* 2021-03-09T08:52:45.814 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-03-09T08:52:45.816 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-09T08:52:45.838 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:45 +0000 (0:00:00.063) 0:02:40.859 ********* 2021-03-09T08:52:45.878 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:45.901 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:45 +0000 (0:00:00.063) 0:02:40.923 ********* 2021-03-09T08:52:46.066 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-03-09T08:52:46.068 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-09T08:52:46.090 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:46 +0000 (0:00:00.188) 0:02:41.112 ********* 2021-03-09T08:52:46.269 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-03-09T08:52:46.271 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-09T08:52:46.294 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:46 +0000 (0:00:00.204) 0:02:41.316 ********* 2021-03-09T08:52:46.323 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-03-09T08:52:46.325 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:46.347 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:46 +0000 (0:00:00.052) 0:02:41.369 ********* 2021-03-09T08:52:46.386 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-03-09T08:52:46.388 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:46.398 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:46.400 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi146.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-03-09T08:52:46.430 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:46 +0000 (0:00:00.081) 0:02:41.451 ********* 2021-03-09T08:52:46.489 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi146.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:46.532 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:46.544 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:46.555 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:46.556 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:46.583 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:46 +0000 (0:00:00.153) 0:02:41.605 ********* 2021-03-09T08:52:46.640 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:46.642 INFO:teuthology.task.ansible.out:TASK [testnode : set_fact] ***************************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:46.701 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:46 +0000 (0:00:00.118) 0:02:41.723 ********* 2021-03-09T08:52:46.744 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:46.770 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:46 +0000 (0:00:00.069) 0:02:41.792 ********* 2021-03-09T08:52:46.812 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:46.849 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:46 +0000 (0:00:00.077) 0:02:41.870 ********* 2021-03-09T08:52:46.883 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-03-09T08:52:46.885 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:46.909 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:46 +0000 (0:00:00.060) 0:02:41.931 ********* 2021-03-09T08:52:46.947 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-03-09T08:52:46.949 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:46.976 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:46 +0000 (0:00:00.066) 0:02:41.998 ********* 2021-03-09T08:52:47.006 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-09T08:52:47.008 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:47.034 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:47 +0000 (0:00:00.058) 0:02:42.056 ********* 2021-03-09T08:52:47.064 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-03-09T08:52:47.065 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:47.094 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:47 +0000 (0:00:00.059) 0:02:42.115 ********* 2021-03-09T08:52:47.124 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-09T08:52:47.126 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:47.150 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:47 +0000 (0:00:00.055) 0:02:42.171 ********* 2021-03-09T08:52:47.409 INFO:teuthology.task.ansible.out: 2021-03-09T08:52:47.411 INFO:teuthology.task.ansible.out:TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi146.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010438", "end": "2021-03-09 08:52:47.350426", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-09 08:52:47.339988", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:47.584 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006063", "end": "2021-03-09 08:52:47.551590", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-09 08:52:47.545527", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:47.731 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005482", "end": "2021-03-09 08:52:47.705164", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-09 08:52:47.699682", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:47.920 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.026683", "end": "2021-03-09 08:52:47.881204", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-09 08:52:47.854521", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:47.949 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:47 +0000 (0:00:00.799) 0:02:42.971 ********* 2021-03-09T08:52:47.978 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-09T08:52:48.001 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:47 +0000 (0:00:00.051) 0:02:43.023 ********* 2021-03-09T08:52:48.194 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-03-09T08:52:48.196 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-03-09T08:52:48.272 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-03-09T08:52:48.294 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:48 +0000 (0:00:00.292) 0:02:43.316 ********* 2021-03-09T08:52:48.322 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:48 +0000 (0:00:00.027) 0:02:43.344 ********* 2021-03-09T08:52:49.273 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-03-09T08:52:49.275 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] 2021-03-09T08:52:49.343 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:49 +0000 (0:00:01.021) 0:02:44.365 ********* 2021-03-09T08:52:49.355 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] 2021-03-09T08:52:49.386 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:52:49.388 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-03-09T08:52:49.409 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:49 +0000 (0:00:00.065) 0:02:44.431 ********* 2021-03-09T08:52:49.432 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:49 +0000 (0:00:00.023) 0:02:44.454 ********* 2021-03-09T08:52:49.449 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-03-09T08:52:49.473 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:49 +0000 (0:00:00.040) 0:02:44.495 ********* 2021-03-09T08:52:49.480 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:52:49.482 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:49.506 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:49 +0000 (0:00:00.033) 0:02:44.528 ********* 2021-03-09T08:52:49.514 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:49.541 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:49 +0000 (0:00:00.034) 0:02:44.563 ********* 2021-03-09T08:52:49.560 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-03-09T08:52:49.562 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-03-09T08:52:49.590 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:49 +0000 (0:00:00.048) 0:02:44.612 ********* 2021-03-09T08:52:49.597 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-03-09T08:52:49.624 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:49 +0000 (0:00:00.034) 0:02:44.646 ********* 2021-03-09T08:52:50.301 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-09T08:52:50.302 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"cache_update_time": 1615279794, "cache_updated": false, "changed": false} 2021-03-09T08:52:50.325 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:50 +0000 (0:00:00.701) 0:02:45.347 ********* 2021-03-09T08:52:50.355 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:52:50.357 INFO:teuthology.task.ansible.out:skipping: [smithi146.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:50.383 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:50 +0000 (0:00:00.057) 0:02:45.405 ********* 2021-03-09T08:52:50.391 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-09T08:52:50.393 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"cache_update_time": 1615279787, "cache_updated": false, "changed": false} 2021-03-09T08:52:50.419 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:50 +0000 (0:00:00.035) 0:02:45.440 ********* 2021-03-09T08:52:50.425 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:52:50.427 INFO:teuthology.task.ansible.out:ok: [smithi146.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-03-09T08:52:50.453 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:50 +0000 (0:00:00.034) 0:02:45.475 ********* 2021-03-09T08:52:50.460 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:52:50.461 INFO:teuthology.task.ansible.out:skipping: [smithi005.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:52:50.488 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:50 +0000 (0:00:00.034) 0:02:45.510 ********* 2021-03-09T08:52:50.530 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:52:50.532 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-03-09T08:52:50.556 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:50 +0000 (0:00:00.068) 0:02:45.578 ********* 2021-03-09T08:52:50.673 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-03-09T08:52:50.675 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:52:50.791 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:52:50.823 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:52:50.853 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:50 +0000 (0:00:00.296) 0:02:45.875 ********* 2021-03-09T08:52:50.947 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:52:50.986 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:50 +0000 (0:00:00.132) 0:02:46.008 ********* 2021-03-09T08:52:51.308 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-03-09T08:52:51.311 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279970.9158359-25004-277342853039428/source", "state": "file", "uid": 0} 2021-03-09T08:52:51.456 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279971.05213-25015-91074224663823/source", "state": "file", "uid": 0} 2021-03-09T08:52:51.703 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279971.3376198-25004-254967185964557/source", "state": "file", "uid": 0} 2021-03-09T08:52:51.738 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:51 +0000 (0:00:00.752) 0:02:46.760 ********* 2021-03-09T08:52:51.904 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279971.4958236-25015-207732763720073/source", "state": "file", "uid": 0} 2021-03-09T08:52:51.932 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:51 +0000 (0:00:00.194) 0:02:46.954 ********* 2021-03-09T08:52:56.402 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-09T08:52:56.404 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": " Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-_va1q4ru\n", "stderr_lines": [" Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-_va1q4ru"], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-_va1q4ru\nCollecting docopt\n Downloading docopt-0.6.2.tar.gz (25 kB)\nCollecting toml\n Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)\nBuilding wheels for collected packages: registries-conf-ctl, docopt\n Building wheel for registries-conf-ctl (setup.py): started\n Building wheel for registries-conf-ctl (setup.py): finished with status 'done'\n Created wheel for registries-conf-ctl: filename=registries_conf_ctl-1.0-py3-none-any.whl size=6035 sha256=26aaa449e6d3b6d462a7ecf40f6dcad603dd0fe12e1c1a50d28faa2facc5c24e\n Stored in directory: /tmp/pip-ephem-wheel-cache-8wxirmaj/wheels/66/49/be/0ca4a7322f6cd17532451655790e478b4f82776818b44d6d42\n Building wheel for docopt (setup.py): started\n Building wheel for docopt (setup.py): finished with status 'done'\n Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13704 sha256=4b346237763c3297b7f1a1a37f453bbfc791a706b4f210a7a7c0110884216720\n Stored in directory: /root/.cache/pip/wheels/56/ea/58/ead137b087d9e326852a851351d1debf4ada529b6ac0ec4e8c\nSuccessfully built registries-conf-ctl docopt\nInstalling collected packages: docopt, toml, registries-conf-ctl\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github. 2021-03-09T08:52:56.406 INFO:teuthology.task.ansible.out:com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-_va1q4ru", "Collecting docopt", " Downloading docopt-0.6.2.tar.gz (25 kB)", "Collecting toml", " Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)", "Building wheels for collected packages: registries-conf-ctl, docopt", " Building wheel for registries-conf-ctl (setup.py): started", " Building wheel for registries-conf-ctl (setup.py): finished with status 'done'", " Created wheel for registries-conf-ctl: filename=registries_conf_ctl-1.0-py3-none-any.whl size=6035 sha256=26aaa449e6d3b6d462a7ecf40f6dcad603dd0fe12e1c1a50d28faa2facc5c24e", " Stored in directory: /tmp/pip-ephem-wheel-cache-8wxirmaj/wheels/66/49/be/0ca4a7322f6cd17532451655790e478b4f82776818b44d6d42", " Building wheel for docopt (setup.py): started", " Building wheel for docopt (setup.py): finished with status 'done'", " Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13704 sha256=4b346237763c3297b7f1a1a37f453bbfc791a706b4f210a7a7c0110884216720", " Stored in directory: /root/.cache/pip/wheels/56/ea/58/ead137b087d9e326852a851351d1debf4ada529b6ac0ec4e8c", "Successfully built registries-conf-ctl docopt", "Installing collected packages: docopt, toml, registries-conf-ctl", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-03-09T08:52:56.430 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:56 +0000 (0:00:04.497) 0:02:51.452 ********* 2021-03-09T08:52:56.620 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* ok: [smithi146.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-09T08:52:56.646 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:56 +0000 (0:00:00.216) 0:02:51.668 ********* 2021-03-09T08:52:56.798 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-09T08:52:56.800 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": " Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-njrrnhv0\n", "stderr_lines": [" Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-njrrnhv0"], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-njrrnhv0\nCollecting docopt\n Downloading docopt-0.6.2.tar.gz (25 kB)\nCollecting toml\n Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)\nBuilding wheels for collected packages: registries-conf-ctl, docopt\n Building wheel for registries-conf-ctl (setup.py): started\n Building wheel for registries-conf-ctl (setup.py): finished with status 'done'\n Created wheel for registries-conf-ctl: filename=registries_conf_ctl-1.0-py3-none-any.whl size=6035 sha256=26aaa449e6d3b6d462a7ecf40f6dcad603dd0fe12e1c1a50d28faa2facc5c24e\n Stored in directory: /tmp/pip-ephem-wheel-cache-m8lyt8al/wheels/66/49/be/0ca4a7322f6cd17532451655790e478b4f82776818b44d6d42\n Building wheel for docopt (setup.py): started\n Building wheel for docopt (setup.py): finished with status 'done'\n Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13704 sha256=4b346237763c3297b7f1a1a37f453bbfc791a706b4f210a7a7c0110884216720\n Stored in directory: /root/.cache/pip/wheels/56/ea/58/ead137b087d9e326852a851351d1debf4ada529b6ac0ec4e8c\nSuccessfully built registries-conf-ctl docopt\nInstalling collected packages: docopt, toml, registries-conf-ctl\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github. 2021-03-09T08:52:56.802 INFO:teuthology.task.ansible.out:com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-njrrnhv0", "Collecting docopt", " Downloading docopt-0.6.2.tar.gz (25 kB)", "Collecting toml", " Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)", "Building wheels for collected packages: registries-conf-ctl, docopt", " Building wheel for registries-conf-ctl (setup.py): started", " Building wheel for registries-conf-ctl (setup.py): finished with status 'done'", " Created wheel for registries-conf-ctl: filename=registries_conf_ctl-1.0-py3-none-any.whl size=6035 sha256=26aaa449e6d3b6d462a7ecf40f6dcad603dd0fe12e1c1a50d28faa2facc5c24e", " Stored in directory: /tmp/pip-ephem-wheel-cache-m8lyt8al/wheels/66/49/be/0ca4a7322f6cd17532451655790e478b4f82776818b44d6d42", " Building wheel for docopt (setup.py): started", " Building wheel for docopt (setup.py): finished with status 'done'", " Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13704 sha256=4b346237763c3297b7f1a1a37f453bbfc791a706b4f210a7a7c0110884216720", " Stored in directory: /root/.cache/pip/wheels/56/ea/58/ead137b087d9e326852a851351d1debf4ada529b6ac0ec4e8c", "Successfully built registries-conf-ctl docopt", "Installing collected packages: docopt, toml, registries-conf-ctl", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-03-09T08:52:56.826 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:56 +0000 (0:00:00.179) 0:02:51.848 ********* 2021-03-09T08:52:57.035 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-03-09T08:52:57.036 INFO:teuthology.task.ansible.out:ok: [smithi005.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-09T08:52:57.060 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:57 +0000 (0:00:00.233) 0:02:52.081 ********* 2021-03-09T08:52:57.099 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-03-09T08:52:57.101 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279976.7068138-25551-143493828762480/source", "state": "file", "uid": 0} 2021-03-09T08:52:57.128 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:57 +0000 (0:00:00.068) 0:02:52.150 ********* 2021-03-09T08:52:57.346 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-03-09T08:52:57.348 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.038435", "end": "2021-03-09 08:52:57.323613", "rc": 0, "start": "2021-03-09 08:52:57.285178", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:57.375 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:57 +0000 (0:00:00.246) 0:02:52.397 ********* 2021-03-09T08:52:57.529 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-03-09T08:52:57.531 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279977.119727-25613-58021212363197/source", "state": "file", "uid": 0} 2021-03-09T08:52:57.555 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:57 +0000 (0:00:00.180) 0:02:52.577 ********* 2021-03-09T08:52:57.779 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-03-09T08:52:57.780 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.037222", "end": "2021-03-09 08:52:57.756116", "rc": 0, "start": "2021-03-09 08:52:57.718894", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:52:57.807 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:52:57 +0000 (0:00:00.251) 0:02:52.829 ********* 2021-03-09T08:53:01.165 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-03-09T08:53:01.166 INFO:teuthology.task.ansible.out:changed: [smithi146.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target docker.socket network-online.target containerd.service systemd-journald.socket firewalld.service sysinit.target system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "yes", "DelegateControllers": "cpu cpuacct cpuset io blkio memory devices pids bpf-fir 2021-03-09T08:53:01.167 INFO:teuthology.task.ansible.out:ewall bpf-devices", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", 2021-03-09T08:53:01.169 INFO:teuthology.task.ansible.out:"LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127781", "LimitSIGPENDINGSoft": "127781", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMPolicy": "continue", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", 2021-03-09T08:53:01.171 INFO:teuthology.task.ansible.out: "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "docker.socket sysinit.target system.slice", "Restart": "always", "RestartKillSignal": "15", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:49:37 UTC", "StateChangeTimestampMonotonic": "43318925", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutAbortUSec": "infinity", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-09T08:53:01.632 INFO:teuthology.task.ansible.out:changed: [smithi005.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "systemd-journald.socket docker.socket system.slice basic.target sysinit.target network-online.target containerd.service firewalld.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "yes", "DelegateControllers": "cpu cpuacct cpuset io blkio memory devices pids bpf-fir 2021-03-09T08:53:01.634 INFO:teuthology.task.ansible.out:ewall bpf-devices", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", 2021-03-09T08:53:01.636 INFO:teuthology.task.ansible.out:"LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127805", "LimitSIGPENDINGSoft": "127805", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMPolicy": "continue", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", 2021-03-09T08:53:01.637 INFO:teuthology.task.ansible.out: "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice docker.socket sysinit.target", "Restart": "always", "RestartKillSignal": "15", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:48:38 UTC", "StateChangeTimestampMonotonic": "32644389", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutAbortUSec": "infinity", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-09T08:53:01.710 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-03-09T08:53:01.712 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi005.front.sepia.ceph.com : ok=129 changed=35 unreachable=0 failed=0 skipped=182 rescued=0 ignored=0 smithi146.front.sepia.ceph.com : ok=129 changed=35 unreachable=0 failed=0 skipped=182 rescued=0 ignored=0 Tuesday 09 March 2021 08:53:01 +0000 (0:00:03.905) 0:02:56.734 ********* =============================================================================== 2021-03-09T08:53:01.713 INFO:teuthology.task.ansible.out:Create all admin users with sudo access. ------------------------------- 20.73s users : Update authorized_keys using the keys repo ---------------------- 7.93s container-host : Install registries-conf-ctl ---------------------------- 4.50s container-host : Restart docker service --------------------------------- 3.91s testnode : Ensure packages are not present. ----------------------------- 2.44s testnode : Update apt cache. -------------------------------------------- 2.28s Gathering Facts --------------------------------------------------------- 2.25s testnode : Install apt keys --------------------------------------------- 2.06s testnode : Install packages via pip ------------------------------------- 2.04s testnode : Install packages --------------------------------------------- 1.79s users : Clone the keys repo --------------------------------------------- 1.26s Gathering Facts --------------------------------------------------------- 1.02s Gathering Facts --------------------------------------------------------- 0.93s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.92s testnode : List any leftover Ceph artifacts from previous jobs ---------- 0.80s ansible-managed : Add authorized keys for the ansible user. ------------- 0.80s testnode : Make sure apt dependencies are installed --------------------- 0.76s container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert --- 0.75s testnode : Upgrade packages --------------------------------------------- 0.75s users : Check keys_repo HEAD sha1 --------------------------------------- 0.74s 2021-03-09T08:53:02.088 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi005.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2021-03-09T08:53:02.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T08:53:02.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T08:53:02.140 DEBUG:teuthology.orchestra.run.smithi005:> true 2021-03-09T08:53:02.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T08:53:02.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T08:53:02.309 DEBUG:teuthology.orchestra.run.smithi146:> true 2021-03-09T08:53:02.424 INFO:teuthology.run_tasks:Running task clock... 2021-03-09T08:53:02.440 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-03-09T08:53:02.440 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-09T08:53:02.442 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-09T08:53:02.480 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Mar 08:53:02 ntpd[8289]: ntpd 4.2.8p12@1.3728-o (1): Starting 2021-03-09T08:53:02.480 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Mar 08:53:02 ntpd[8289]: Command line: ntpd -gq 2021-03-09T08:53:02.481 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Mar 08:53:02 ntpd[8289]: proto: precision = 0.123 usec (-23) 2021-03-09T08:53:02.482 INFO:teuthology.orchestra.run.smithi005.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-09T08:53:02.482 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Mar 08:53:02 ntpd[8289]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-09T08:53:02.484 INFO:teuthology.orchestra.run.smithi005.stderr:restrict ::: KOD does nothing without LIMITED. 2021-03-09T08:53:02.485 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Mar 08:53:02 ntpd[8289]: restrict ::: KOD does nothing without LIMITED. 2021-03-09T08:53:02.485 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Mar 08:53:02 ntpd[8289]: Listen and drop on 0 v6wildcard [::]:123 2021-03-09T08:53:02.486 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Mar 08:53:02 ntpd[8289]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-03-09T08:53:02.486 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Mar 08:53:02 ntpd[8289]: Listen normally on 2 lo 127.0.0.1:123 2021-03-09T08:53:02.486 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Mar 08:53:02 ntpd[8289]: Listen normally on 3 ens1f0 172.21.15.5:123 2021-03-09T08:53:02.486 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Mar 08:53:02 ntpd[8289]: Listen normally on 4 lo [::1]:123 2021-03-09T08:53:02.487 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Mar 08:53:02 ntpd[8289]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1442%4]:123 2021-03-09T08:53:02.487 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Mar 08:53:02 ntpd[8289]: Listening on routing socket on fd #22 for interface updates 2021-03-09T08:53:03.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:03 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:03.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:03 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:03.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:03 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:03.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:03 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:03.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:03 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:03.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:03 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:03.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:03 ntpd[8289]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:03.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:03 ntpd[8289]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:03.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:03 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:03.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:03 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:03.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:03 ntpd[8289]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:03.485 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:03 ntpd[8289]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:04.481 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:04 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:04.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:04 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:04.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:04 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:04.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:04 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:04.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:04 ntpd[8289]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:04.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:04 ntpd[8289]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:05.486 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:05 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:05.486 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:05 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:05.486 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:05 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:05.487 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:05 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:05.487 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:05 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:05.487 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:05 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:05.487 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:05 ntpd[8289]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:05.488 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:05 ntpd[8289]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:05.488 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:05 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:05.488 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:05 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:05.488 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:05 ntpd[8289]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:05.488 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:05 ntpd[8289]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:06.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:06 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:06.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:06 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:06.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:06 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:06.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:06 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:06.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:06 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:06.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:06 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:06.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:06 ntpd[8289]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:06.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:06 ntpd[8289]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:06.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:06 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:06.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:06 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:06.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:06 ntpd[8289]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:06.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:06 ntpd[8289]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:07.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:07 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:07.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:07 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:07.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:07 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:07.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:07 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:07.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:07 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:07.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:07 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:07.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:07 ntpd[8289]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:07.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:07 ntpd[8289]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:07.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:07 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:07.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:07 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:07.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:07 ntpd[8289]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:07.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:07 ntpd[8289]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:08.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:08 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:08.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:08 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:08.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:08 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:08.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:08 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:08.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:08 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:08.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:08 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:08.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:08 ntpd[8289]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:08.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:08 ntpd[8289]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:08.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:08 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:08.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:08 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:08.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:08 ntpd[8289]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:08.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:08 ntpd[8289]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:09.481 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:09 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:09.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:09 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:09.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:09 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:09.482 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:09 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:09.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:09 ntpd[8289]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:09.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:09 ntpd[8289]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:09.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:09 ntpd[8289]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:09.483 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:09 ntpd[8289]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:09.484 INFO:teuthology.orchestra.run.smithi005.stdout: 9 Mar 08:53:09 ntpd[8289]: ntpd: time slew +0.000887 s 2021-03-09T08:53:09.484 INFO:teuthology.orchestra.run.smithi005.stdout:ntpd: time slew +0.000887s 2021-03-09T08:53:09.484 INFO:teuthology.orchestra.run.smithi005.stderr: 9 Mar 08:53:09 ntpd[8289]: can't open /var/log/ntpstats/loopstats.20210309: Permission denied 2021-03-09T08:53:09.542 INFO:teuthology.orchestra.run.smithi005.stdout: remote refid st t when poll reach delay offset jitter 2021-03-09T08:53:09.542 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================== 2021-03-09T08:53:09.542 INFO:teuthology.orchestra.run.smithi005.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T08:53:09.542 INFO:teuthology.orchestra.run.smithi005.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T08:53:09.543 INFO:teuthology.orchestra.run.smithi005.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T08:53:09.543 INFO:teuthology.orchestra.run.smithi005.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T08:53:09.543 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-09T08:53:09.545 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-09T08:53:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: 9 Mar 08:53:09 ntpd[8490]: ntpd 4.2.8p12@1.3728-o (1): Starting 2021-03-09T08:53:09.582 INFO:teuthology.orchestra.run.smithi146.stdout: 9 Mar 08:53:09 ntpd[8490]: Command line: ntpd -gq 2021-03-09T08:53:09.583 INFO:teuthology.orchestra.run.smithi146.stdout: 9 Mar 08:53:09 ntpd[8490]: proto: precision = 0.109 usec (-23) 2021-03-09T08:53:09.583 INFO:teuthology.orchestra.run.smithi146.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-09T08:53:09.584 INFO:teuthology.orchestra.run.smithi146.stdout: 9 Mar 08:53:09 ntpd[8490]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-09T08:53:09.584 INFO:teuthology.orchestra.run.smithi146.stderr:restrict ::: KOD does nothing without LIMITED. 2021-03-09T08:53:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: 9 Mar 08:53:09 ntpd[8490]: restrict ::: KOD does nothing without LIMITED. 2021-03-09T08:53:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: 9 Mar 08:53:09 ntpd[8490]: Listen and drop on 0 v6wildcard [::]:123 2021-03-09T08:53:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: 9 Mar 08:53:09 ntpd[8490]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-03-09T08:53:09.586 INFO:teuthology.orchestra.run.smithi146.stdout: 9 Mar 08:53:09 ntpd[8490]: Listen normally on 2 lo 127.0.0.1:123 2021-03-09T08:53:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: 9 Mar 08:53:09 ntpd[8490]: Listen normally on 3 enp3s0f1 172.21.15.146:123 2021-03-09T08:53:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: 9 Mar 08:53:09 ntpd[8490]: Listen normally on 4 lo [::1]:123 2021-03-09T08:53:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: 9 Mar 08:53:09 ntpd[8490]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:76fd%5]:123 2021-03-09T08:53:09.587 INFO:teuthology.orchestra.run.smithi146.stdout: 9 Mar 08:53:09 ntpd[8490]: Listening on routing socket on fd #22 for interface updates 2021-03-09T08:53:10.583 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:10 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:10.583 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:10 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:10.583 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:10 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:10.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:10 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:10.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:10 ntpd[8490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:10.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:10 ntpd[8490]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:11.583 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:11 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:11.583 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:11 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:11.583 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:11 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:11.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:11 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:11.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:11 ntpd[8490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:11.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:11 ntpd[8490]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:12.583 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:12 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:12.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:12 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:12.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:12 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:12.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:12 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:12.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:12 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:12.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:12 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:12.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:12 ntpd[8490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:12.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:12 ntpd[8490]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:12.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:12 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:12.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:12 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:12.586 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:12 ntpd[8490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:12.586 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:12 ntpd[8490]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:13.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:13 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:13.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:13 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:13.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:13 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:13.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:13 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:13.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:13 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:13.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:13 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:13.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:13 ntpd[8490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:13.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:13 ntpd[8490]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:13.586 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:13 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:13.586 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:13 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:13.587 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:13 ntpd[8490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:13.589 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:13 ntpd[8490]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:14.583 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:14 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:14.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:14 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:14.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:14 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:14.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:14 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:14.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:14 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:14.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:14 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:14.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:14 ntpd[8490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:14.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:14 ntpd[8490]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:14.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:14 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:14.586 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:14 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:14.586 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:14 ntpd[8490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:14.586 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:14 ntpd[8490]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:15.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:15 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:15.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:15 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:15.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:15 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:15.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:15 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:15.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:15 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:15.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:15 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:15.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:15 ntpd[8490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:15.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:15 ntpd[8490]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:15.586 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:15 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:15.586 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:15 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:15.586 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:15 ntpd[8490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:15.587 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:15 ntpd[8490]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:16.583 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:16 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:16.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:16 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:16.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:16 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:16.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:16 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:16.584 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:16 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:16.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:16 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:16.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:16 ntpd[8490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:16.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:16 ntpd[8490]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:16.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:16 ntpd[8490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T08:53:16.585 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:16 ntpd[8490]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T08:53:16.586 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:16 ntpd[8490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T08:53:16.586 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:16 ntpd[8490]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T08:53:16.586 INFO:teuthology.orchestra.run.smithi146.stdout: 9 Mar 08:53:16 ntpd[8490]: ntpd: time slew +0.000734 s 2021-03-09T08:53:16.587 INFO:teuthology.orchestra.run.smithi146.stdout:ntpd: time slew +0.000734s 2021-03-09T08:53:16.587 INFO:teuthology.orchestra.run.smithi146.stderr: 9 Mar 08:53:16 ntpd[8490]: can't open /var/log/ntpstats/loopstats.20210309: Permission denied 2021-03-09T08:53:16.639 INFO:teuthology.orchestra.run.smithi146.stdout: remote refid st t when poll reach delay offset jitter 2021-03-09T08:53:16.640 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================== 2021-03-09T08:53:16.640 INFO:teuthology.orchestra.run.smithi146.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T08:53:16.640 INFO:teuthology.orchestra.run.smithi146.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T08:53:16.640 INFO:teuthology.orchestra.run.smithi146.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T08:53:16.641 INFO:teuthology.orchestra.run.smithi146.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T08:53:16.642 INFO:teuthology.run_tasks:Running task install... 2021-03-09T08:53:16.652 DEBUG:teuthology.task.install:project ceph 2021-03-09T08:53:16.656 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}} 2021-03-09T08:53:16.657 DEBUG:teuthology.task.install:config {'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'} 2021-03-09T08:53:16.658 INFO:teuthology.task.install:Using flavor: basic 2021-03-09T08:53:16.668 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-03-09T08:53:16.670 INFO:teuthology.task.install:extra packages: [] 2021-03-09T08:53:16.670 DEBUG:teuthology.orchestra.run.smithi005:> sudo lsb_release -is 2021-03-09T08:53:16.723 INFO:teuthology.orchestra.run.smithi005.stdout:Ubuntu 2021-03-09T08:53:16.728 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T08:53:16.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo lsb_release -is 2021-03-09T08:53:16.730 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-key list | grep Ceph 2021-03-09T08:53:16.782 INFO:teuthology.orchestra.run.smithi146.stdout:Ubuntu 2021-03-09T08:53:16.788 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T08:53:16.788 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-key list | grep Ceph 2021-03-09T08:53:16.838 INFO:teuthology.orchestra.run.smithi005.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-09T08:53:16.889 INFO:teuthology.orchestra.run.smithi146.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-09T08:53:16.899 INFO:teuthology.orchestra.run.smithi005.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-09T08:53:16.900 INFO:teuthology.orchestra.run.smithi005.stdout:uid [ unknown] Ceph.com (release key) 2021-03-09T08:53:16.901 DEBUG:teuthology.orchestra.run.smithi005:> sudo lsb_release -is 2021-03-09T08:53:16.948 INFO:teuthology.orchestra.run.smithi146.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-09T08:53:16.948 INFO:teuthology.orchestra.run.smithi146.stdout:uid [ unknown] Ceph.com (release key) 2021-03-09T08:53:16.949 INFO:teuthology.orchestra.run.smithi005.stdout:Ubuntu 2021-03-09T08:53:16.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo lsb_release -is 2021-03-09T08:53:16.953 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T08:53:16.954 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-03-09T08:53:16.954 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:53:17.001 INFO:teuthology.orchestra.run.smithi146.stdout:Ubuntu 2021-03-09T08:53:17.004 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T08:53:17.004 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-03-09T08:53:17.005 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:53:17.121 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default/ 2021-03-09T08:53:17.122 INFO:teuthology.task.install.deb:Package version is 17.0.0-1742-g89fb622b-1focal 2021-03-09T08:53:17.157 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default/ 2021-03-09T08:53:17.158 INFO:teuthology.task.install.deb:Package version is 17.0.0-1742-g89fb622b-1focal 2021-03-09T08:53:17.241 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2021-03-09T08:53:17.242 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-09T08:53:17.258 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2021-03-09T08:53:17.274 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2021-03-09T08:53:17.275 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-09T08:53:17.292 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get update 2021-03-09T08:53:17.492 INFO:teuthology.orchestra.run.smithi146.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal InRelease 2021-03-09T08:53:17.511 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2021-03-09T08:53:17.512 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2021-03-09T08:53:17.513 INFO:teuthology.orchestra.run.smithi005.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal InRelease 2021-03-09T08:53:17.515 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal Release [7,680 B] 2021-03-09T08:53:17.535 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal Release [7,680 B] 2021-03-09T08:53:17.540 INFO:teuthology.orchestra.run.smithi146.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal Release.gpg 2021-03-09T08:53:17.542 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2021-03-09T08:53:17.555 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease 2021-03-09T08:53:17.561 INFO:teuthology.orchestra.run.smithi005.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal Release.gpg 2021-03-09T08:53:17.564 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 Packages [16.6 kB] 2021-03-09T08:53:17.585 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 Packages [16.6 kB] 2021-03-09T08:53:17.596 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main i386 Packages [3,099 B] 2021-03-09T08:53:17.619 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main i386 Packages [3,099 B] 2021-03-09T08:53:17.622 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2021-03-09T08:53:17.634 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2021-03-09T08:53:17.702 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2021-03-09T08:53:17.714 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2021-03-09T08:53:18.156 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 27.3 kB in 1s (38.6 kB/s) 2021-03-09T08:53:18.178 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 27.3 kB in 1s (38.3 kB/s) 2021-03-09T08:53:18.883 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2021-03-09T08:53:18.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-1742-g89fb622b-1focal cephadm=17.0.0-1742-g89fb622b-1focal ceph-mds=17.0.0-1742-g89fb622b-1focal ceph-mgr=17.0.0-1742-g89fb622b-1focal ceph-common=17.0.0-1742-g89fb622b-1focal ceph-fuse=17.0.0-1742-g89fb622b-1focal ceph-test=17.0.0-1742-g89fb622b-1focal radosgw=17.0.0-1742-g89fb622b-1focal python3-rados=17.0.0-1742-g89fb622b-1focal python3-rgw=17.0.0-1742-g89fb622b-1focal python3-cephfs=17.0.0-1742-g89fb622b-1focal python3-rbd=17.0.0-1742-g89fb622b-1focal libcephfs2=17.0.0-1742-g89fb622b-1focal libcephfs-dev=17.0.0-1742-g89fb622b-1focal librados2=17.0.0-1742-g89fb622b-1focal librbd1=17.0.0-1742-g89fb622b-1focal rbd-fuse=17.0.0-1742-g89fb622b-1focal 2021-03-09T08:53:18.914 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2021-03-09T08:53:18.936 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-1742-g89fb622b-1focal cephadm=17.0.0-1742-g89fb622b-1focal ceph-mds=17.0.0-1742-g89fb622b-1focal ceph-mgr=17.0.0-1742-g89fb622b-1focal ceph-common=17.0.0-1742-g89fb622b-1focal ceph-fuse=17.0.0-1742-g89fb622b-1focal ceph-test=17.0.0-1742-g89fb622b-1focal radosgw=17.0.0-1742-g89fb622b-1focal python3-rados=17.0.0-1742-g89fb622b-1focal python3-rgw=17.0.0-1742-g89fb622b-1focal python3-cephfs=17.0.0-1742-g89fb622b-1focal python3-rbd=17.0.0-1742-g89fb622b-1focal libcephfs2=17.0.0-1742-g89fb622b-1focal libcephfs-dev=17.0.0-1742-g89fb622b-1focal librados2=17.0.0-1742-g89fb622b-1focal librbd1=17.0.0-1742-g89fb622b-1focal rbd-fuse=17.0.0-1742-g89fb622b-1focal 2021-03-09T08:53:18.967 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2021-03-09T08:53:19.003 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2021-03-09T08:53:19.119 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2021-03-09T08:53:19.120 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2021-03-09T08:53:19.143 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2021-03-09T08:53:19.144 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2021-03-09T08:53:19.245 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages were automatically installed and are no longer required: 2021-03-09T08:53:19.248 INFO:teuthology.orchestra.run.smithi146.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2021-03-09T08:53:19.248 INFO:teuthology.orchestra.run.smithi146.stdout:Use 'sudo apt autoremove' to remove them. 2021-03-09T08:53:19.248 INFO:teuthology.orchestra.run.smithi146.stdout:The following additional packages will be installed: 2021-03-09T08:53:19.249 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-03-09T08:53:19.249 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-03-09T08:53:19.249 INFO:teuthology.orchestra.run.smithi146.stdout: libjaeger libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2021-03-09T08:53:19.249 INFO:teuthology.orchestra.run.smithi146.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 2021-03-09T08:53:19.250 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2021-03-09T08:53:19.250 INFO:teuthology.orchestra.run.smithi146.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2021-03-09T08:53:19.250 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2021-03-09T08:53:19.251 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2021-03-09T08:53:19.251 INFO:teuthology.orchestra.run.smithi146.stdout: python3-html5lib python3-jinja2 python3-joblib python3-json-pointer 2021-03-09T08:53:19.251 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-03-09T08:53:19.251 INFO:teuthology.orchestra.run.smithi146.stdout: python3-lxml python3-mako python3-markupsafe python3-oauthlib 2021-03-09T08:53:19.252 INFO:teuthology.orchestra.run.smithi146.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2021-03-09T08:53:19.252 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2021-03-09T08:53:19.252 INFO:teuthology.orchestra.run.smithi146.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2021-03-09T08:53:19.252 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyinotify python3-pytest python3-repoze.lru 2021-03-09T08:53:19.253 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2021-03-09T08:53:19.253 INFO:teuthology.orchestra.run.smithi146.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-03-09T08:53:19.253 INFO:teuthology.orchestra.run.smithi146.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2021-03-09T08:53:19.253 INFO:teuthology.orchestra.run.smithi146.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2021-03-09T08:53:19.254 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket python3-webtest python3-werkzeug socat xmlstarlet 2021-03-09T08:53:19.254 INFO:teuthology.orchestra.run.smithi146.stdout:Suggested packages: 2021-03-09T08:53:19.254 INFO:teuthology.orchestra.run.smithi146.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2021-03-09T08:53:19.254 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2021-03-09T08:53:19.255 INFO:teuthology.orchestra.run.smithi146.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2021-03-09T08:53:19.255 INFO:teuthology.orchestra.run.smithi146.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2021-03-09T08:53:19.255 INFO:teuthology.orchestra.run.smithi146.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2021-03-09T08:53:19.256 INFO:teuthology.orchestra.run.smithi146.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2021-03-09T08:53:19.256 INFO:teuthology.orchestra.run.smithi146.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2021-03-09T08:53:19.257 INFO:teuthology.orchestra.run.smithi146.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2021-03-09T08:53:19.257 INFO:teuthology.orchestra.run.smithi146.stdout:Recommended packages: 2021-03-09T08:53:19.257 INFO:teuthology.orchestra.run.smithi146.stdout: btrfs-tools 2021-03-09T08:53:19.266 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages were automatically installed and are no longer required: 2021-03-09T08:53:19.268 INFO:teuthology.orchestra.run.smithi005.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2021-03-09T08:53:19.269 INFO:teuthology.orchestra.run.smithi005.stdout:Use 'sudo apt autoremove' to remove them. 2021-03-09T08:53:19.270 INFO:teuthology.orchestra.run.smithi005.stdout:The following additional packages will be installed: 2021-03-09T08:53:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-03-09T08:53:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-03-09T08:53:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: libjaeger libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2021-03-09T08:53:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 2021-03-09T08:53:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2021-03-09T08:53:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2021-03-09T08:53:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2021-03-09T08:53:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2021-03-09T08:53:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: python3-html5lib python3-jinja2 python3-joblib python3-json-pointer 2021-03-09T08:53:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-03-09T08:53:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: python3-lxml python3-mako python3-markupsafe python3-oauthlib 2021-03-09T08:53:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2021-03-09T08:53:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2021-03-09T08:53:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2021-03-09T08:53:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyinotify python3-pytest python3-repoze.lru 2021-03-09T08:53:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2021-03-09T08:53:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-03-09T08:53:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2021-03-09T08:53:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2021-03-09T08:53:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: python3-websocket python3-webtest python3-werkzeug socat xmlstarlet 2021-03-09T08:53:19.275 INFO:teuthology.orchestra.run.smithi005.stdout:Suggested packages: 2021-03-09T08:53:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2021-03-09T08:53:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2021-03-09T08:53:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2021-03-09T08:53:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2021-03-09T08:53:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2021-03-09T08:53:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2021-03-09T08:53:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2021-03-09T08:53:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2021-03-09T08:53:19.277 INFO:teuthology.orchestra.run.smithi005.stdout:Recommended packages: 2021-03-09T08:53:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: btrfs-tools 2021-03-09T08:53:19.337 INFO:teuthology.orchestra.run.smithi146.stdout:The following NEW packages will be installed: 2021-03-09T08:53:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-03-09T08:53:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-03-09T08:53:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-03-09T08:53:19.338 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libjs-sphinxdoc libjs-underscore 2021-03-09T08:53:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: liblbfgsb0 libleveldb1d liblttng-ust0 liblua5.3-0 liboath0 libonig5 2021-03-09T08:53:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq4 libradosstriper1 librdkafka1 librgw2 libxslt1.1 nvme-cli 2021-03-09T08:53:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2021-03-09T08:53:19.339 INFO:teuthology.orchestra.run.smithi146.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2021-03-09T08:53:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2021-03-09T08:53:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2021-03-09T08:53:19.340 INFO:teuthology.orchestra.run.smithi146.stdout: python3-joblib python3-json-pointer python3-jsonpatch python3-jwt 2021-03-09T08:53:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2021-03-09T08:53:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2021-03-09T08:53:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2021-03-09T08:53:19.341 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2021-03-09T08:53:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-rados 2021-03-09T08:53:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd python3-repoze.lru python3-requests-oauthlib python3-rgw 2021-03-09T08:53:19.342 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes python3-rsa python3-scipy python3-simplegeneric 2021-03-09T08:53:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: python3-simplejson python3-singledispatch python3-sklearn 2021-03-09T08:53:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2021-03-09T08:53:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2021-03-09T08:53:19.343 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-03-09T08:53:19.344 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages will be upgraded: 2021-03-09T08:53:19.344 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 librbd1 2021-03-09T08:53:19.363 INFO:teuthology.orchestra.run.smithi005.stdout:The following NEW packages will be installed: 2021-03-09T08:53:19.364 INFO:teuthology.orchestra.run.smithi005.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-03-09T08:53:19.364 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-03-09T08:53:19.364 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-03-09T08:53:19.364 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libjs-sphinxdoc libjs-underscore 2021-03-09T08:53:19.365 INFO:teuthology.orchestra.run.smithi005.stdout: liblbfgsb0 libleveldb1d liblttng-ust0 liblua5.3-0 liboath0 libonig5 2021-03-09T08:53:19.366 INFO:teuthology.orchestra.run.smithi005.stdout: librabbitmq4 libradosstriper1 librdkafka1 librgw2 libxslt1.1 nvme-cli 2021-03-09T08:53:19.366 INFO:teuthology.orchestra.run.smithi005.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2021-03-09T08:53:19.366 INFO:teuthology.orchestra.run.smithi005.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2021-03-09T08:53:19.366 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2021-03-09T08:53:19.367 INFO:teuthology.orchestra.run.smithi005.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2021-03-09T08:53:19.367 INFO:teuthology.orchestra.run.smithi005.stdout: python3-joblib python3-json-pointer python3-jsonpatch python3-jwt 2021-03-09T08:53:19.367 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2021-03-09T08:53:19.367 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2021-03-09T08:53:19.368 INFO:teuthology.orchestra.run.smithi005.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2021-03-09T08:53:19.368 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2021-03-09T08:53:19.368 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-rados 2021-03-09T08:53:19.368 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd python3-repoze.lru python3-requests-oauthlib python3-rgw 2021-03-09T08:53:19.368 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes python3-rsa python3-scipy python3-simplegeneric 2021-03-09T08:53:19.369 INFO:teuthology.orchestra.run.smithi005.stdout: python3-simplejson python3-singledispatch python3-sklearn 2021-03-09T08:53:19.369 INFO:teuthology.orchestra.run.smithi005.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2021-03-09T08:53:19.369 INFO:teuthology.orchestra.run.smithi005.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2021-03-09T08:53:19.369 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-03-09T08:53:19.369 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages will be upgraded: 2021-03-09T08:53:19.370 INFO:teuthology.orchestra.run.smithi005.stdout: librados2 librbd1 2021-03-09T08:53:19.469 INFO:teuthology.orchestra.run.smithi146.stdout:2 upgraded, 101 newly installed, 0 to remove and 7 not upgraded. 2021-03-09T08:53:19.470 INFO:teuthology.orchestra.run.smithi146.stdout:Need to get 171 MB of archives. 2021-03-09T08:53:19.470 INFO:teuthology.orchestra.run.smithi146.stdout:After this operation, 792 MB of additional disk space will be used. 2021-03-09T08:53:19.470 INFO:teuthology.orchestra.run.smithi146.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 libjaeger amd64 17.0.0-1742-g89fb622b-1focal [3,724 B] 2021-03-09T08:53:19.500 INFO:teuthology.orchestra.run.smithi005.stdout:2 upgraded, 101 newly installed, 0 to remove and 7 not upgraded. 2021-03-09T08:53:19.500 INFO:teuthology.orchestra.run.smithi005.stdout:Need to get 171 MB of archives. 2021-03-09T08:53:19.500 INFO:teuthology.orchestra.run.smithi005.stdout:After this operation, 792 MB of additional disk space will be used. 2021-03-09T08:53:19.501 INFO:teuthology.orchestra.run.smithi005.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 libjaeger amd64 17.0.0-1742-g89fb622b-1focal [3,724 B] 2021-03-09T08:53:19.517 INFO:teuthology.orchestra.run.smithi146.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.0.0-1742-g89fb622b-1focal [3,280 kB] 2021-03-09T08:53:19.522 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2021-03-09T08:53:19.547 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.0.0-1742-g89fb622b-1focal [3,280 kB] 2021-03-09T08:53:19.548 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2021-03-09T08:53:19.750 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.0.0-1742-g89fb622b-1focal [3,351 kB] 2021-03-09T08:53:19.877 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.0.0-1742-g89fb622b-1focal [3,351 kB] 2021-03-09T08:53:19.904 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2021-03-09T08:53:19.905 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2021-03-09T08:53:19.905 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2021-03-09T08:53:19.909 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2021-03-09T08:53:19.911 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2021-03-09T08:53:19.914 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.0.0-1742-g89fb622b-1focal [706 kB] 2021-03-09T08:53:19.915 INFO:teuthology.orchestra.run.smithi146.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2021-03-09T08:53:19.944 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2021-03-09T08:53:19.958 INFO:teuthology.orchestra.run.smithi146.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.0.0-1742-g89fb622b-1focal [409 kB] 2021-03-09T08:53:19.982 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2021-03-09T08:53:19.993 INFO:teuthology.orchestra.run.smithi146.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.0.0-1742-g89fb622b-1focal [21.7 kB] 2021-03-09T08:53:19.994 INFO:teuthology.orchestra.run.smithi146.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.0.0-1742-g89fb622b-1focal [198 kB] 2021-03-09T08:53:20.000 INFO:teuthology.orchestra.run.smithi146.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.0.0-1742-g89fb622b-1focal [29.4 kB] 2021-03-09T08:53:20.001 INFO:teuthology.orchestra.run.smithi146.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.0.0-1742-g89fb622b-1focal [385 kB] 2021-03-09T08:53:20.014 INFO:teuthology.orchestra.run.smithi146.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.0.0-1742-g89fb622b-1focal [3,595 kB] 2021-03-09T08:53:20.077 INFO:teuthology.orchestra.run.smithi146.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2021-03-09T08:53:20.082 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2021-03-09T08:53:20.104 INFO:teuthology.orchestra.run.smithi005.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2021-03-09T08:53:20.110 INFO:teuthology.orchestra.run.smithi146.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2021-03-09T08:53:20.111 INFO:teuthology.orchestra.run.smithi005.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2021-03-09T08:53:20.118 INFO:teuthology.orchestra.run.smithi146.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2021-03-09T08:53:20.140 INFO:teuthology.orchestra.run.smithi005.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2021-03-09T08:53:20.143 INFO:teuthology.orchestra.run.smithi005.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2021-03-09T08:53:20.144 INFO:teuthology.orchestra.run.smithi005.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2021-03-09T08:53:20.145 INFO:teuthology.orchestra.run.smithi146.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2021-03-09T08:53:20.147 INFO:teuthology.orchestra.run.smithi146.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2021-03-09T08:53:20.150 INFO:teuthology.orchestra.run.smithi146.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2021-03-09T08:53:20.158 INFO:teuthology.orchestra.run.smithi005.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2021-03-09T08:53:20.160 INFO:teuthology.orchestra.run.smithi005.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2021-03-09T08:53:20.161 INFO:teuthology.orchestra.run.smithi005.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2021-03-09T08:53:20.162 INFO:teuthology.orchestra.run.smithi005.stdout:Get:18 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.0.0-1742-g89fb622b-1focal [706 kB] 2021-03-09T08:53:20.163 INFO:teuthology.orchestra.run.smithi146.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2021-03-09T08:53:20.164 INFO:teuthology.orchestra.run.smithi146.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2021-03-09T08:53:20.165 INFO:teuthology.orchestra.run.smithi146.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2021-03-09T08:53:20.167 INFO:teuthology.orchestra.run.smithi146.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2021-03-09T08:53:20.203 INFO:teuthology.orchestra.run.smithi146.stdout:Get:26 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.0.0-1742-g89fb622b-1focal [116 kB] 2021-03-09T08:53:20.206 INFO:teuthology.orchestra.run.smithi146.stdout:Get:27 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.0.0-1742-g89fb622b-1focal [448 kB] 2021-03-09T08:53:20.219 INFO:teuthology.orchestra.run.smithi005.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2021-03-09T08:53:20.222 INFO:teuthology.orchestra.run.smithi005.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2021-03-09T08:53:20.223 INFO:teuthology.orchestra.run.smithi005.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2021-03-09T08:53:20.224 INFO:teuthology.orchestra.run.smithi005.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2021-03-09T08:53:20.225 INFO:teuthology.orchestra.run.smithi005.stdout:Get:23 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.0.0-1742-g89fb622b-1focal [409 kB] 2021-03-09T08:53:20.238 INFO:teuthology.orchestra.run.smithi146.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2021-03-09T08:53:20.239 INFO:teuthology.orchestra.run.smithi146.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2021-03-09T08:53:20.240 INFO:teuthology.orchestra.run.smithi146.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2021-03-09T08:53:20.243 INFO:teuthology.orchestra.run.smithi146.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2021-03-09T08:53:20.245 INFO:teuthology.orchestra.run.smithi146.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2021-03-09T08:53:20.247 INFO:teuthology.orchestra.run.smithi146.stdout:Get:33 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.0.0-1742-g89fb622b-1focal [22.2 MB] 2021-03-09T08:53:20.281 INFO:teuthology.orchestra.run.smithi005.stdout:Get:24 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.0.0-1742-g89fb622b-1focal [21.7 kB] 2021-03-09T08:53:20.282 INFO:teuthology.orchestra.run.smithi005.stdout:Get:25 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.0.0-1742-g89fb622b-1focal [198 kB] 2021-03-09T08:53:20.288 INFO:teuthology.orchestra.run.smithi005.stdout:Get:26 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.0.0-1742-g89fb622b-1focal [29.4 kB] 2021-03-09T08:53:20.290 INFO:teuthology.orchestra.run.smithi005.stdout:Get:27 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.0.0-1742-g89fb622b-1focal [385 kB] 2021-03-09T08:53:20.297 INFO:teuthology.orchestra.run.smithi005.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2021-03-09T08:53:20.298 INFO:teuthology.orchestra.run.smithi005.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2021-03-09T08:53:20.301 INFO:teuthology.orchestra.run.smithi005.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-waitress all 1.4.1-1 [80.6 kB] 2021-03-09T08:53:20.302 INFO:teuthology.orchestra.run.smithi005.stdout:Get:31 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.0.0-1742-g89fb622b-1focal [3,595 kB] 2021-03-09T08:53:20.304 INFO:teuthology.orchestra.run.smithi005.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2021-03-09T08:53:20.305 INFO:teuthology.orchestra.run.smithi005.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2021-03-09T08:53:20.317 INFO:teuthology.orchestra.run.smithi146.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-waitress all 1.4.1-1 [80.6 kB] 2021-03-09T08:53:20.320 INFO:teuthology.orchestra.run.smithi146.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2021-03-09T08:53:20.321 INFO:teuthology.orchestra.run.smithi146.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2021-03-09T08:53:20.323 INFO:teuthology.orchestra.run.smithi005.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2021-03-09T08:53:20.341 INFO:teuthology.orchestra.run.smithi146.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2021-03-09T08:53:20.341 INFO:teuthology.orchestra.run.smithi146.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2021-03-09T08:53:20.375 INFO:teuthology.orchestra.run.smithi005.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2021-03-09T08:53:20.376 INFO:teuthology.orchestra.run.smithi005.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2021-03-09T08:53:20.377 INFO:teuthology.orchestra.run.smithi005.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2021-03-09T08:53:20.380 INFO:teuthology.orchestra.run.smithi005.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2021-03-09T08:53:20.397 INFO:teuthology.orchestra.run.smithi146.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2021-03-09T08:53:20.397 INFO:teuthology.orchestra.run.smithi146.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2021-03-09T08:53:20.400 INFO:teuthology.orchestra.run.smithi146.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2021-03-09T08:53:20.409 INFO:teuthology.orchestra.run.smithi146.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2021-03-09T08:53:20.412 INFO:teuthology.orchestra.run.smithi146.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2021-03-09T08:53:20.453 INFO:teuthology.orchestra.run.smithi005.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2021-03-09T08:53:20.457 INFO:teuthology.orchestra.run.smithi005.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2021-03-09T08:53:20.457 INFO:teuthology.orchestra.run.smithi005.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-underscore all 1.9.1~dfsg-1 [98.6 kB] 2021-03-09T08:53:20.461 INFO:teuthology.orchestra.run.smithi005.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2021-03-09T08:53:20.465 INFO:teuthology.orchestra.run.smithi005.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2021-03-09T08:53:20.468 INFO:teuthology.orchestra.run.smithi005.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2021-03-09T08:53:20.476 INFO:teuthology.orchestra.run.smithi146.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-underscore all 1.9.1~dfsg-1 [98.6 kB] 2021-03-09T08:53:20.480 INFO:teuthology.orchestra.run.smithi146.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2021-03-09T08:53:20.486 INFO:teuthology.orchestra.run.smithi146.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2021-03-09T08:53:20.487 INFO:teuthology.orchestra.run.smithi146.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2021-03-09T08:53:20.488 INFO:teuthology.orchestra.run.smithi146.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2021-03-09T08:53:20.531 INFO:teuthology.orchestra.run.smithi005.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2021-03-09T08:53:20.532 INFO:teuthology.orchestra.run.smithi005.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2021-03-09T08:53:20.555 INFO:teuthology.orchestra.run.smithi146.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2021-03-09T08:53:20.695 INFO:teuthology.orchestra.run.smithi005.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.0.0-1742-g89fb622b-1focal [116 kB] 2021-03-09T08:53:20.698 INFO:teuthology.orchestra.run.smithi005.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.0.0-1742-g89fb622b-1focal [448 kB] 2021-03-09T08:53:20.705 INFO:teuthology.orchestra.run.smithi005.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.0.0-1742-g89fb622b-1focal [22.2 MB] 2021-03-09T08:53:20.825 INFO:teuthology.orchestra.run.smithi005.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2021-03-09T08:53:20.893 INFO:teuthology.orchestra.run.smithi146.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2021-03-09T08:53:20.900 INFO:teuthology.orchestra.run.smithi005.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2021-03-09T08:53:20.902 INFO:teuthology.orchestra.run.smithi005.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2021-03-09T08:53:20.961 INFO:teuthology.orchestra.run.smithi146.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2021-03-09T08:53:20.966 INFO:teuthology.orchestra.run.smithi146.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2021-03-09T08:53:20.970 INFO:teuthology.orchestra.run.smithi005.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2021-03-09T08:53:20.970 INFO:teuthology.orchestra.run.smithi005.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2021-03-09T08:53:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2021-03-09T08:53:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2021-03-09T08:53:20.972 INFO:teuthology.orchestra.run.smithi005.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2021-03-09T08:53:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2021-03-09T08:53:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2021-03-09T08:53:20.981 INFO:teuthology.orchestra.run.smithi005.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2021-03-09T08:53:20.982 INFO:teuthology.orchestra.run.smithi005.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2021-03-09T08:53:21.005 INFO:teuthology.orchestra.run.smithi146.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2021-03-09T08:53:21.005 INFO:teuthology.orchestra.run.smithi146.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2021-03-09T08:53:21.006 INFO:teuthology.orchestra.run.smithi146.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2021-03-09T08:53:21.006 INFO:teuthology.orchestra.run.smithi146.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2021-03-09T08:53:21.007 INFO:teuthology.orchestra.run.smithi146.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2021-03-09T08:53:21.007 INFO:teuthology.orchestra.run.smithi146.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2021-03-09T08:53:21.007 INFO:teuthology.orchestra.run.smithi146.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2021-03-09T08:53:21.060 INFO:teuthology.orchestra.run.smithi005.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2021-03-09T08:53:21.065 INFO:teuthology.orchestra.run.smithi005.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-json-pointer all 2.0-0ubuntu1 [8,320 B] 2021-03-09T08:53:21.065 INFO:teuthology.orchestra.run.smithi005.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jsonpatch all 1.23-3 [12.0 kB] 2021-03-09T08:53:21.066 INFO:teuthology.orchestra.run.smithi005.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2021-03-09T08:53:21.068 INFO:teuthology.orchestra.run.smithi146.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2021-03-09T08:53:21.069 INFO:teuthology.orchestra.run.smithi005.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2021-03-09T08:53:21.070 INFO:teuthology.orchestra.run.smithi146.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2021-03-09T08:53:21.072 INFO:teuthology.orchestra.run.smithi005.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2021-03-09T08:53:21.073 INFO:teuthology.orchestra.run.smithi005.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2021-03-09T08:53:21.076 INFO:teuthology.orchestra.run.smithi005.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2021-03-09T08:53:21.138 INFO:teuthology.orchestra.run.smithi005.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2021-03-09T08:53:21.143 INFO:teuthology.orchestra.run.smithi005.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2021-03-09T08:53:21.147 INFO:teuthology.orchestra.run.smithi146.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2021-03-09T08:53:21.151 INFO:teuthology.orchestra.run.smithi146.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-json-pointer all 2.0-0ubuntu1 [8,320 B] 2021-03-09T08:53:21.152 INFO:teuthology.orchestra.run.smithi146.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jsonpatch all 1.23-3 [12.0 kB] 2021-03-09T08:53:21.152 INFO:teuthology.orchestra.run.smithi146.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2021-03-09T08:53:21.155 INFO:teuthology.orchestra.run.smithi146.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2021-03-09T08:53:21.158 INFO:teuthology.orchestra.run.smithi146.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2021-03-09T08:53:21.159 INFO:teuthology.orchestra.run.smithi146.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2021-03-09T08:53:21.162 INFO:teuthology.orchestra.run.smithi146.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2021-03-09T08:53:21.216 INFO:teuthology.orchestra.run.smithi005.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2021-03-09T08:53:21.217 INFO:teuthology.orchestra.run.smithi005.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2021-03-09T08:53:21.218 INFO:teuthology.orchestra.run.smithi005.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2021-03-09T08:53:21.218 INFO:teuthology.orchestra.run.smithi005.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2021-03-09T08:53:21.221 INFO:teuthology.orchestra.run.smithi005.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.2 [1,382 kB] 2021-03-09T08:53:21.226 INFO:teuthology.orchestra.run.smithi146.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2021-03-09T08:53:21.231 INFO:teuthology.orchestra.run.smithi146.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2021-03-09T08:53:21.240 INFO:teuthology.orchestra.run.smithi005.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2021-03-09T08:53:21.241 INFO:teuthology.orchestra.run.smithi005.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2021-03-09T08:53:21.241 INFO:teuthology.orchestra.run.smithi005.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2021-03-09T08:53:21.295 INFO:teuthology.orchestra.run.smithi005.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2021-03-09T08:53:21.298 INFO:teuthology.orchestra.run.smithi005.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-py all 1.8.1-1 [65.4 kB] 2021-03-09T08:53:21.306 INFO:teuthology.orchestra.run.smithi146.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2021-03-09T08:53:21.307 INFO:teuthology.orchestra.run.smithi146.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2021-03-09T08:53:21.308 INFO:teuthology.orchestra.run.smithi146.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2021-03-09T08:53:21.308 INFO:teuthology.orchestra.run.smithi146.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2021-03-09T08:53:21.311 INFO:teuthology.orchestra.run.smithi146.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.2 [1,382 kB] 2021-03-09T08:53:21.330 INFO:teuthology.orchestra.run.smithi146.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2021-03-09T08:53:21.331 INFO:teuthology.orchestra.run.smithi146.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2021-03-09T08:53:21.331 INFO:teuthology.orchestra.run.smithi146.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2021-03-09T08:53:21.372 INFO:teuthology.orchestra.run.smithi005.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2021-03-09T08:53:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2021-03-09T08:53:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2021-03-09T08:53:21.377 INFO:teuthology.orchestra.run.smithi005.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2021-03-09T08:53:21.391 INFO:teuthology.orchestra.run.smithi146.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2021-03-09T08:53:21.396 INFO:teuthology.orchestra.run.smithi146.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-py all 1.8.1-1 [65.4 kB] 2021-03-09T08:53:21.469 INFO:teuthology.orchestra.run.smithi146.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2021-03-09T08:53:21.527 INFO:teuthology.orchestra.run.smithi146.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2021-03-09T08:53:21.529 INFO:teuthology.orchestra.run.smithi146.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2021-03-09T08:53:21.530 INFO:teuthology.orchestra.run.smithi146.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.0.0-1742-g89fb622b-1focal [5,873 kB] 2021-03-09T08:53:21.532 INFO:teuthology.orchestra.run.smithi146.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2021-03-09T08:53:21.777 INFO:teuthology.orchestra.run.smithi146.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.0.0-1742-g89fb622b-1focal [183 kB] 2021-03-09T08:53:21.783 INFO:teuthology.orchestra.run.smithi146.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.0.0-1742-g89fb622b-1focal [1,489 kB] 2021-03-09T08:53:21.859 INFO:teuthology.orchestra.run.smithi146.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.0.0-1742-g89fb622b-1focal [6,925 kB] 2021-03-09T08:53:22.226 INFO:teuthology.orchestra.run.smithi146.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.0.0-1742-g89fb622b-1focal [26.0 MB] 2021-03-09T08:53:22.548 INFO:teuthology.orchestra.run.smithi005.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.0.0-1742-g89fb622b-1focal [5,873 kB] 2021-03-09T08:53:22.979 INFO:teuthology.orchestra.run.smithi005.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.0.0-1742-g89fb622b-1focal [183 kB] 2021-03-09T08:53:22.983 INFO:teuthology.orchestra.run.smithi005.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.0.0-1742-g89fb622b-1focal [1,489 kB] 2021-03-09T08:53:23.124 INFO:teuthology.orchestra.run.smithi005.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.0.0-1742-g89fb622b-1focal [6,925 kB] 2021-03-09T08:53:23.674 INFO:teuthology.orchestra.run.smithi005.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.0.0-1742-g89fb622b-1focal [26.0 MB] 2021-03-09T08:53:23.731 INFO:teuthology.orchestra.run.smithi146.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.0.0-1742-g89fb622b-1focal [3,816 B] 2021-03-09T08:53:23.733 INFO:teuthology.orchestra.run.smithi146.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.0.0-1742-g89fb622b-1focal [810 kB] 2021-03-09T08:53:23.746 INFO:teuthology.orchestra.run.smithi146.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.0.0-1742-g89fb622b-1focal [2,302 kB] 2021-03-09T08:53:23.823 INFO:teuthology.orchestra.run.smithi146.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.0.0-1742-g89fb622b-1focal [63.1 kB] 2021-03-09T08:53:23.825 INFO:teuthology.orchestra.run.smithi146.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.0.0-1742-g89fb622b-1focal [68.9 kB] 2021-03-09T08:53:23.827 INFO:teuthology.orchestra.run.smithi146.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.0.0-1742-g89fb622b-1focal [2,332 kB] 2021-03-09T08:53:23.939 INFO:teuthology.orchestra.run.smithi146.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.0.0-1742-g89fb622b-1focal [7,892 kB] 2021-03-09T08:53:24.383 INFO:teuthology.orchestra.run.smithi146.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.0.0-1742-g89fb622b-1focal [3,932 B] 2021-03-09T08:53:24.386 INFO:teuthology.orchestra.run.smithi146.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-rook all 17.0.0-1742-g89fb622b-1focal [22.4 kB] 2021-03-09T08:53:24.387 INFO:teuthology.orchestra.run.smithi146.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.0.0-1742-g89fb622b-1focal [48.8 MB] 2021-03-09T08:53:25.802 INFO:teuthology.orchestra.run.smithi005.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.0.0-1742-g89fb622b-1focal [3,816 B] 2021-03-09T08:53:25.802 INFO:teuthology.orchestra.run.smithi005.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.0.0-1742-g89fb622b-1focal [810 kB] 2021-03-09T08:53:25.815 INFO:teuthology.orchestra.run.smithi005.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.0.0-1742-g89fb622b-1focal [2,302 kB] 2021-03-09T08:53:25.938 INFO:teuthology.orchestra.run.smithi005.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.0.0-1742-g89fb622b-1focal [63.1 kB] 2021-03-09T08:53:25.940 INFO:teuthology.orchestra.run.smithi005.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.0.0-1742-g89fb622b-1focal [68.9 kB] 2021-03-09T08:53:25.942 INFO:teuthology.orchestra.run.smithi005.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.0.0-1742-g89fb622b-1focal [2,332 kB] 2021-03-09T08:53:26.130 INFO:teuthology.orchestra.run.smithi005.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.0.0-1742-g89fb622b-1focal [7,892 kB] 2021-03-09T08:53:26.653 INFO:teuthology.orchestra.run.smithi005.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.0.0-1742-g89fb622b-1focal [3,932 B] 2021-03-09T08:53:26.654 INFO:teuthology.orchestra.run.smithi005.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-rook all 17.0.0-1742-g89fb622b-1focal [22.4 kB] 2021-03-09T08:53:26.655 INFO:teuthology.orchestra.run.smithi005.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.0.0-1742-g89fb622b-1focal [48.8 MB] 2021-03-09T08:53:27.519 INFO:teuthology.orchestra.run.smithi146.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.0.0-1742-g89fb622b-1focal [19.9 kB] 2021-03-09T08:53:27.520 INFO:teuthology.orchestra.run.smithi146.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.0.0-1742-g89fb622b-1focal [11.1 MB] 2021-03-09T08:53:28.052 INFO:teuthology.orchestra.run.smithi146.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.0.0-1742-g89fb622b-1focal [77.6 kB] 2021-03-09T08:53:28.630 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 171 MB in 9s (19.6 MB/s) 2021-03-09T08:53:28.736 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libjaeger. 2021-03-09T08:53:28.776 INFO:teuthology.orchestra.run.smithi146.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101691 files and directories currently installed.) 2021-03-09T08:53:28.779 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../000-libjaeger_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:28.818 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libjaeger (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:29.161 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-03-09T08:53:29.173 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../001-liblttng-ust0_2.11.0-1_amd64.deb ... 2021-03-09T08:53:29.214 INFO:teuthology.orchestra.run.smithi005.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.0.0-1742-g89fb622b-1focal [19.9 kB] 2021-03-09T08:53:29.215 INFO:teuthology.orchestra.run.smithi005.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.0.0-1742-g89fb622b-1focal [11.1 MB] 2021-03-09T08:53:29.255 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2021-03-09T08:53:29.461 INFO:teuthology.orchestra.run.smithi005.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.0.0-1742-g89fb622b-1focal [77.6 kB] 2021-03-09T08:53:29.571 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../002-librbd1_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:29.687 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librbd1 (17.0.0-1742-g89fb622b-1focal) over (15.2.8-0ubuntu0.20.04.1) ... 2021-03-09T08:53:30.050 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 171 MB in 10s (17.0 MB/s) 2021-03-09T08:53:30.143 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjaeger. 2021-03-09T08:53:30.185 INFO:teuthology.orchestra.run.smithi005.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101691 files and directories currently installed.) 2021-03-09T08:53:30.188 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../000-libjaeger_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:30.217 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjaeger (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:30.342 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../003-librados2_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:30.441 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librados2 (17.0.0-1742-g89fb622b-1focal) over (15.2.8-0ubuntu0.20.04.1) ... 2021-03-09T08:53:30.569 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-03-09T08:53:30.583 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../001-liblttng-ust0_2.11.0-1_amd64.deb ... 2021-03-09T08:53:30.671 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2021-03-09T08:53:31.053 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../002-librbd1_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:31.103 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libcephfs2. 2021-03-09T08:53:31.117 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../004-libcephfs2_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:31.162 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libcephfs2 (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:31.171 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librbd1 (17.0.0-1742-g89fb622b-1focal) over (15.2.8-0ubuntu0.20.04.1) ... 2021-03-09T08:53:31.471 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rados. 2021-03-09T08:53:31.485 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../005-python3-rados_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:31.529 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rados (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:31.829 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../003-librados2_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:32.160 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librados2 (17.0.0-1742-g89fb622b-1focal) over (15.2.8-0ubuntu0.20.04.1) ... 2021-03-09T08:53:32.161 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-03-09T08:53:32.161 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../006-python3-ceph-argparse_17.0.0-1742-g89fb622b-1focal_all.deb ... 2021-03-09T08:53:32.161 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-ceph-argparse (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:32.350 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cephfs. 2021-03-09T08:53:32.366 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../007-python3-cephfs_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:32.409 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cephfs (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:32.758 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-ceph-common. 2021-03-09T08:53:32.774 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../008-python3-ceph-common_17.0.0-1742-g89fb622b-1focal_all.deb ... 2021-03-09T08:53:32.804 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libcephfs2. 2021-03-09T08:53:32.809 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-ceph-common (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:32.819 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../004-libcephfs2_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:32.855 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libcephfs2 (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:33.129 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-prettytable. 2021-03-09T08:53:33.142 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-5_all.deb ... 2021-03-09T08:53:33.179 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2021-03-09T08:53:33.188 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rados. 2021-03-09T08:53:33.202 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../005-python3-rados_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:33.240 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rados (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:33.429 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rbd. 2021-03-09T08:53:33.442 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../010-python3-rbd_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:33.507 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rbd (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:33.592 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-03-09T08:53:33.604 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../006-python3-ceph-argparse_17.0.0-1742-g89fb622b-1focal_all.deb ... 2021-03-09T08:53:33.642 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-ceph-argparse (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:33.866 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2021-03-09T08:53:33.882 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../011-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2021-03-09T08:53:33.908 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cephfs. 2021-03-09T08:53:33.921 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../007-python3-cephfs_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:33.924 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2021-03-09T08:53:33.959 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cephfs (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:34.280 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-03-09T08:53:34.293 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../012-librabbitmq4_0.10.0-1_amd64.deb ... 2021-03-09T08:53:34.309 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-ceph-common. 2021-03-09T08:53:34.323 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../008-python3-ceph-common_17.0.0-1742-g89fb622b-1focal_all.deb ... 2021-03-09T08:53:34.364 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2021-03-09T08:53:34.368 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-ceph-common (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:34.671 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-prettytable. 2021-03-09T08:53:34.684 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-5_all.deb ... 2021-03-09T08:53:34.722 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2021-03-09T08:53:34.725 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-03-09T08:53:34.741 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../013-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2021-03-09T08:53:34.784 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2021-03-09T08:53:35.000 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rbd. 2021-03-09T08:53:35.014 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../010-python3-rbd_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:35.050 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rbd (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:35.125 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librgw2. 2021-03-09T08:53:35.138 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../014-librgw2_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:35.174 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librgw2 (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:35.392 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2021-03-09T08:53:35.406 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../011-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2021-03-09T08:53:35.452 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2021-03-09T08:53:35.764 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rgw. 2021-03-09T08:53:35.777 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../015-python3-rgw_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:35.795 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-03-09T08:53:35.810 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../012-librabbitmq4_0.10.0-1_amd64.deb ... 2021-03-09T08:53:35.814 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rgw (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:35.880 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2021-03-09T08:53:36.134 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libleveldb1d:amd64. 2021-03-09T08:53:36.150 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../016-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2021-03-09T08:53:36.182 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-03-09T08:53:36.192 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2021-03-09T08:53:36.197 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../013-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2021-03-09T08:53:36.233 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2021-03-09T08:53:36.540 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liboath0. 2021-03-09T08:53:36.553 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../017-liboath0_2.6.1-1.3_amd64.deb ... 2021-03-09T08:53:36.598 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2021-03-09T08:53:36.599 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librgw2. 2021-03-09T08:53:36.614 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../014-librgw2_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:36.649 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librgw2 (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:36.893 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libradosstriper1. 2021-03-09T08:53:36.909 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../018-libradosstriper1_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:36.969 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libradosstriper1 (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:37.233 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rgw. 2021-03-09T08:53:37.246 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../015-python3-rgw_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:37.275 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rgw (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:37.346 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-common. 2021-03-09T08:53:37.360 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../019-ceph-common_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:37.407 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-common (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:37.608 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libleveldb1d:amd64. 2021-03-09T08:53:37.623 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../016-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2021-03-09T08:53:37.659 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2021-03-09T08:53:37.980 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liboath0. 2021-03-09T08:53:37.996 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../017-liboath0_2.6.1-1.3_amd64.deb ... 2021-03-09T08:53:38.063 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2021-03-09T08:53:38.353 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libradosstriper1. 2021-03-09T08:53:38.367 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../018-libradosstriper1_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:38.413 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libradosstriper1 (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:38.876 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-common. 2021-03-09T08:53:38.890 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../019-ceph-common_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:38.930 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-common (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:39.570 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-base. 2021-03-09T08:53:39.583 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../020-ceph-base_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:39.622 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-base (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:40.576 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cryptography. 2021-03-09T08:53:40.589 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../021-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2021-03-09T08:53:40.617 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2021-03-09T08:53:40.987 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-openssl. 2021-03-09T08:53:40.999 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../022-python3-openssl_19.0.0-1build1_all.deb ... 2021-03-09T08:53:41.040 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2021-03-09T08:53:41.210 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-base. 2021-03-09T08:53:41.223 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../020-ceph-base_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:41.271 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-base (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:41.323 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-03-09T08:53:41.336 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../023-ceph-mgr-modules-core_17.0.0-1742-g89fb622b-1focal_all.deb ... 2021-03-09T08:53:41.382 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-modules-core (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:41.734 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-bcrypt. 2021-03-09T08:53:41.750 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../024-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2021-03-09T08:53:41.784 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2021-03-09T08:53:42.086 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cherrypy3. 2021-03-09T08:53:42.100 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2021-03-09T08:53:42.161 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2021-03-09T08:53:42.300 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cryptography. 2021-03-09T08:53:42.316 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../021-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2021-03-09T08:53:42.395 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2021-03-09T08:53:42.497 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jwt. 2021-03-09T08:53:42.513 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../026-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2021-03-09T08:53:42.547 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2021-03-09T08:53:42.754 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-openssl. 2021-03-09T08:53:42.769 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../022-python3-openssl_19.0.0-1build1_all.deb ... 2021-03-09T08:53:42.813 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2021-03-09T08:53:42.880 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-logutils. 2021-03-09T08:53:42.893 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2021-03-09T08:53:42.942 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-logutils (0.3.3-7) ... 2021-03-09T08:53:43.106 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-03-09T08:53:43.122 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../023-ceph-mgr-modules-core_17.0.0-1742-g89fb622b-1focal_all.deb ... 2021-03-09T08:53:43.165 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-modules-core (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:43.271 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-markupsafe. 2021-03-09T08:53:43.284 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2021-03-09T08:53:43.329 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2021-03-09T08:53:43.543 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-bcrypt. 2021-03-09T08:53:43.555 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../024-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2021-03-09T08:53:43.594 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2021-03-09T08:53:43.621 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-mako. 2021-03-09T08:53:43.635 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2021-03-09T08:53:43.679 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2021-03-09T08:53:43.887 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cherrypy3. 2021-03-09T08:53:43.905 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2021-03-09T08:53:43.945 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2021-03-09T08:53:44.015 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-simplegeneric. 2021-03-09T08:53:44.028 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2021-03-09T08:53:44.073 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2021-03-09T08:53:44.298 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jwt. 2021-03-09T08:53:44.311 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../026-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2021-03-09T08:53:44.348 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2021-03-09T08:53:44.367 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-singledispatch. 2021-03-09T08:53:44.381 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2021-03-09T08:53:44.417 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2021-03-09T08:53:44.673 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-logutils. 2021-03-09T08:53:44.686 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2021-03-09T08:53:44.758 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-webob. 2021-03-09T08:53:44.759 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-logutils (0.3.3-7) ... 2021-03-09T08:53:44.771 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2021-03-09T08:53:44.815 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2021-03-09T08:53:45.008 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-markupsafe. 2021-03-09T08:53:45.024 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2021-03-09T08:53:45.066 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2021-03-09T08:53:45.150 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-soupsieve. 2021-03-09T08:53:45.164 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2021-03-09T08:53:45.209 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2021-03-09T08:53:45.391 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-mako. 2021-03-09T08:53:45.405 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2021-03-09T08:53:45.441 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2021-03-09T08:53:45.500 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-bs4. 2021-03-09T08:53:45.515 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2021-03-09T08:53:45.578 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2021-03-09T08:53:45.768 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-simplegeneric. 2021-03-09T08:53:45.785 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2021-03-09T08:53:45.827 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2021-03-09T08:53:45.890 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-waitress. 2021-03-09T08:53:45.905 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1_all.deb ... 2021-03-09T08:53:45.945 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-waitress (1.4.1-1) ... 2021-03-09T08:53:46.112 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-singledispatch. 2021-03-09T08:53:46.125 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2021-03-09T08:53:46.171 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2021-03-09T08:53:46.256 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-tempita. 2021-03-09T08:53:46.269 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2021-03-09T08:53:46.307 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-tempita (0.5.2-6) ... 2021-03-09T08:53:46.493 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-webob. 2021-03-09T08:53:46.509 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2021-03-09T08:53:46.552 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2021-03-09T08:53:46.632 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-paste. 2021-03-09T08:53:46.646 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2021-03-09T08:53:46.692 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2021-03-09T08:53:46.880 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-soupsieve. 2021-03-09T08:53:46.892 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2021-03-09T08:53:46.930 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2021-03-09T08:53:47.019 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-03-09T08:53:47.034 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2021-03-09T08:53:47.077 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2021-03-09T08:53:47.222 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-bs4. 2021-03-09T08:53:47.235 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2021-03-09T08:53:47.271 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2021-03-09T08:53:47.444 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pastedeploy. 2021-03-09T08:53:47.456 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2021-03-09T08:53:47.502 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2021-03-09T08:53:47.649 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-waitress. 2021-03-09T08:53:47.663 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1_all.deb ... 2021-03-09T08:53:47.710 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-waitress (1.4.1-1) ... 2021-03-09T08:53:47.928 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-webtest. 2021-03-09T08:53:47.943 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2021-03-09T08:53:47.989 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-webtest (2.0.32-2) ... 2021-03-09T08:53:48.164 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-tempita. 2021-03-09T08:53:48.180 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2021-03-09T08:53:48.225 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-tempita (0.5.2-6) ... 2021-03-09T08:53:48.333 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pecan. 2021-03-09T08:53:48.346 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2021-03-09T08:53:48.384 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2021-03-09T08:53:48.567 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-paste. 2021-03-09T08:53:48.582 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2021-03-09T08:53:48.617 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2021-03-09T08:53:48.719 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-werkzeug. 2021-03-09T08:53:48.731 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2021-03-09T08:53:48.769 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2021-03-09T08:53:48.969 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-03-09T08:53:48.982 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2021-03-09T08:53:49.019 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2021-03-09T08:53:49.170 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr. 2021-03-09T08:53:49.178 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../043-ceph-mgr_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:49.224 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:49.319 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pastedeploy. 2021-03-09T08:53:49.332 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2021-03-09T08:53:49.369 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2021-03-09T08:53:49.660 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mon. 2021-03-09T08:53:49.675 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../044-ceph-mon_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:49.695 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-webtest. 2021-03-09T08:53:49.707 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2021-03-09T08:53:49.718 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mon (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:49.745 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-webtest (2.0.32-2) ... 2021-03-09T08:53:50.067 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pecan. 2021-03-09T08:53:50.080 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2021-03-09T08:53:50.133 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2021-03-09T08:53:50.520 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-werkzeug. 2021-03-09T08:53:50.534 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2021-03-09T08:53:50.595 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2021-03-09T08:53:50.730 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-osd. 2021-03-09T08:53:50.742 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../045-ceph-osd_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:50.805 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-osd (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:50.934 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr. 2021-03-09T08:53:50.947 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../043-ceph-mgr_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:50.992 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:51.480 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mon. 2021-03-09T08:53:51.495 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../044-ceph-mon_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:51.538 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mon (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:53.392 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-osd. 2021-03-09T08:53:53.405 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../045-ceph-osd_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:53.433 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-osd (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:53.685 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph. 2021-03-09T08:53:53.699 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../046-ceph_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:53.735 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:54.037 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-fuse. 2021-03-09T08:53:54.050 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../047-ceph-fuse_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:54.088 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-fuse (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:54.524 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mds. 2021-03-09T08:53:54.539 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../048-ceph-mds_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:54.582 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mds (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:55.085 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package cephadm. 2021-03-09T08:53:55.100 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../049-cephadm_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:55.135 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking cephadm (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:55.446 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jinja2. 2021-03-09T08:53:55.460 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../050-python3-jinja2_2.10.1-2_all.deb ... 2021-03-09T08:53:55.504 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2021-03-09T08:53:55.807 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-03-09T08:53:55.820 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../051-ceph-mgr-cephadm_17.0.0-1742-g89fb622b-1focal_all.deb ... 2021-03-09T08:53:55.857 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-cephadm (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:56.114 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph. 2021-03-09T08:53:56.129 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../046-ceph_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:56.172 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:56.210 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-repoze.lru. 2021-03-09T08:53:56.224 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../052-python3-repoze.lru_0.7-2_all.deb ... 2021-03-09T08:53:56.268 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2021-03-09T08:53:56.491 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-fuse. 2021-03-09T08:53:56.504 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../047-ceph-fuse_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:56.521 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libjs-underscore. 2021-03-09T08:53:56.534 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../053-libjs-underscore_1.9.1~dfsg-1_all.deb ... 2021-03-09T08:53:56.541 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-fuse (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:56.562 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1) ... 2021-03-09T08:53:56.823 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libjs-sphinxdoc. 2021-03-09T08:53:56.836 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../054-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2021-03-09T08:53:56.873 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2021-03-09T08:53:56.928 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mds. 2021-03-09T08:53:56.941 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../048-ceph-mds_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:56.970 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mds (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:57.201 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-routes. 2021-03-09T08:53:57.216 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../055-python3-routes_2.4.1-2_all.deb ... 2021-03-09T08:53:57.259 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-routes (2.4.1-2) ... 2021-03-09T08:53:57.506 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package cephadm. 2021-03-09T08:53:57.519 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../049-cephadm_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:53:57.556 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking cephadm (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:57.579 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-03-09T08:53:57.595 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_17.0.0-1742-g89fb622b-1focal_all.deb ... 2021-03-09T08:53:57.637 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-dashboard (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:57.909 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jinja2. 2021-03-09T08:53:57.921 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../050-python3-jinja2_2.10.1-2_all.deb ... 2021-03-09T08:53:57.959 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2021-03-09T08:53:58.287 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-03-09T08:53:58.300 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../051-ceph-mgr-cephadm_17.0.0-1742-g89fb622b-1focal_all.deb ... 2021-03-09T08:53:58.354 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-cephadm (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:53:58.586 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-decorator. 2021-03-09T08:53:58.601 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../057-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2021-03-09T08:53:58.648 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-repoze.lru. 2021-03-09T08:53:58.662 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../052-python3-repoze.lru_0.7-2_all.deb ... 2021-03-09T08:53:58.669 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2021-03-09T08:53:58.707 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2021-03-09T08:53:58.934 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjs-underscore. 2021-03-09T08:53:58.948 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../053-libjs-underscore_1.9.1~dfsg-1_all.deb ... 2021-03-09T08:53:58.972 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2021-03-09T08:53:58.984 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1) ... 2021-03-09T08:53:58.985 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../058-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2021-03-09T08:53:59.022 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2021-03-09T08:53:59.254 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjs-sphinxdoc. 2021-03-09T08:53:59.267 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../054-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2021-03-09T08:53:59.292 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-scipy. 2021-03-09T08:53:59.304 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2021-03-09T08:53:59.307 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../059-python3-scipy_1.3.3-3build1_amd64.deb ... 2021-03-09T08:53:59.350 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2021-03-09T08:53:59.648 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-routes. 2021-03-09T08:53:59.662 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../055-python3-routes_2.4.1-2_all.deb ... 2021-03-09T08:53:59.706 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-routes (2.4.1-2) ... 2021-03-09T08:54:00.059 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-03-09T08:54:00.069 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_17.0.0-1742-g89fb622b-1focal_all.deb ... 2021-03-09T08:54:00.109 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-dashboard (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:00.683 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-03-09T08:54:00.697 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2021-03-09T08:54:00.741 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2021-03-09T08:54:01.085 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-decorator. 2021-03-09T08:54:01.101 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../057-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2021-03-09T08:54:01.143 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2021-03-09T08:54:01.287 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-joblib. 2021-03-09T08:54:01.301 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../061-python3-joblib_0.14.0-3_all.deb ... 2021-03-09T08:54:01.336 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-joblib (0.14.0-3) ... 2021-03-09T08:54:01.445 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2021-03-09T08:54:01.461 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../058-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2021-03-09T08:54:01.504 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2021-03-09T08:54:01.665 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-sklearn. 2021-03-09T08:54:01.681 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../062-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2021-03-09T08:54:01.723 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2021-03-09T08:54:01.840 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-scipy. 2021-03-09T08:54:01.856 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../059-python3-scipy_1.3.3-3build1_amd64.deb ... 2021-03-09T08:54:01.898 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2021-03-09T08:54:02.423 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-03-09T08:54:02.437 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_17.0.0-1742-g89fb622b-1focal_all.deb ... 2021-03-09T08:54:02.481 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:03.158 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-03-09T08:54:03.172 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2021-03-09T08:54:03.238 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2021-03-09T08:54:03.944 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-joblib. 2021-03-09T08:54:03.960 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../061-python3-joblib_0.14.0-3_all.deb ... 2021-03-09T08:54:04.002 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-joblib (0.14.0-3) ... 2021-03-09T08:54:04.085 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cachetools. 2021-03-09T08:54:04.101 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../064-python3-cachetools_4.0.0-1_all.deb ... 2021-03-09T08:54:04.143 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2021-03-09T08:54:04.390 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-sklearn. 2021-03-09T08:54:04.403 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../062-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2021-03-09T08:54:04.445 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyasn1. 2021-03-09T08:54:04.447 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2021-03-09T08:54:04.460 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../065-python3-pyasn1_0.4.2-3build1_all.deb ... 2021-03-09T08:54:04.504 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2021-03-09T08:54:04.832 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-03-09T08:54:04.848 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../066-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2021-03-09T08:54:04.898 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2021-03-09T08:54:05.115 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-03-09T08:54:05.128 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_17.0.0-1742-g89fb622b-1focal_all.deb ... 2021-03-09T08:54:05.173 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:05.210 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rsa. 2021-03-09T08:54:05.226 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../067-python3-rsa_4.0-3_all.deb ... 2021-03-09T08:54:05.268 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rsa (4.0-3) ... 2021-03-09T08:54:05.571 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-google-auth. 2021-03-09T08:54:05.585 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../068-python3-google-auth_1.5.1-2_all.deb ... 2021-03-09T08:54:05.621 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2021-03-09T08:54:05.951 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-blinker. 2021-03-09T08:54:05.965 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../069-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2021-03-09T08:54:06.008 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2021-03-09T08:54:06.310 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-oauthlib. 2021-03-09T08:54:06.318 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cachetools. 2021-03-09T08:54:06.325 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../070-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2021-03-09T08:54:06.333 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../064-python3-cachetools_4.0.0-1_all.deb ... 2021-03-09T08:54:06.368 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2021-03-09T08:54:06.376 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2021-03-09T08:54:06.713 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyasn1. 2021-03-09T08:54:06.723 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-requests-oauthlib. 2021-03-09T08:54:06.726 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../065-python3-pyasn1_0.4.2-3build1_all.deb ... 2021-03-09T08:54:06.736 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../071-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2021-03-09T08:54:06.771 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2021-03-09T08:54:06.780 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2021-03-09T08:54:07.075 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-websocket. 2021-03-09T08:54:07.091 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../072-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2021-03-09T08:54:07.125 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-03-09T08:54:07.133 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2021-03-09T08:54:07.138 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../066-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2021-03-09T08:54:07.174 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2021-03-09T08:54:07.469 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-kubernetes. 2021-03-09T08:54:07.482 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../073-python3-kubernetes_7.0.0~a1-2_all.deb ... 2021-03-09T08:54:07.485 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rsa. 2021-03-09T08:54:07.501 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../067-python3-rsa_4.0-3_all.deb ... 2021-03-09T08:54:07.519 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2021-03-09T08:54:07.543 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rsa (4.0-3) ... 2021-03-09T08:54:07.855 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-google-auth. 2021-03-09T08:54:07.872 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../068-python3-google-auth_1.5.1-2_all.deb ... 2021-03-09T08:54:07.913 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2021-03-09T08:54:08.007 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-03-09T08:54:08.021 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../074-ceph-mgr-k8sevents_17.0.0-1742-g89fb622b-1focal_all.deb ... 2021-03-09T08:54:08.057 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:08.250 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-blinker. 2021-03-09T08:54:08.265 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../069-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2021-03-09T08:54:08.308 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2021-03-09T08:54:08.402 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-json-pointer. 2021-03-09T08:54:08.416 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../075-python3-json-pointer_2.0-0ubuntu1_all.deb ... 2021-03-09T08:54:08.459 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-json-pointer (2.0-0ubuntu1) ... 2021-03-09T08:54:08.594 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-oauthlib. 2021-03-09T08:54:08.608 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../070-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2021-03-09T08:54:08.644 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2021-03-09T08:54:08.787 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jsonpatch. 2021-03-09T08:54:08.801 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../076-python3-jsonpatch_1.23-3_all.deb ... 2021-03-09T08:54:08.845 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jsonpatch (1.23-3) ... 2021-03-09T08:54:09.007 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-requests-oauthlib. 2021-03-09T08:54:09.022 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../071-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2021-03-09T08:54:09.064 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2021-03-09T08:54:09.149 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-03-09T08:54:09.161 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../077-ceph-mgr-rook_17.0.0-1742-g89fb622b-1focal_all.deb ... 2021-03-09T08:54:09.190 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-rook (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:09.401 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-websocket. 2021-03-09T08:54:09.415 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../072-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2021-03-09T08:54:09.459 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2021-03-09T08:54:09.526 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libonig5:amd64. 2021-03-09T08:54:09.540 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../078-libonig5_6.9.4-1_amd64.deb ... 2021-03-09T08:54:09.584 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2021-03-09T08:54:09.770 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-kubernetes. 2021-03-09T08:54:09.784 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../073-python3-kubernetes_7.0.0~a1-2_all.deb ... 2021-03-09T08:54:09.845 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2021-03-09T08:54:09.962 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libjq1:amd64. 2021-03-09T08:54:09.977 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../079-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2021-03-09T08:54:10.020 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2021-03-09T08:54:10.273 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package jq. 2021-03-09T08:54:10.288 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../080-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2021-03-09T08:54:10.331 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2021-03-09T08:54:10.417 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-03-09T08:54:10.431 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../074-ceph-mgr-k8sevents_17.0.0-1742-g89fb622b-1focal_all.deb ... 2021-03-09T08:54:10.475 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:10.678 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package socat. 2021-03-09T08:54:10.692 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../081-socat_1.7.3.3-2_amd64.deb ... 2021-03-09T08:54:10.745 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-json-pointer. 2021-03-09T08:54:10.752 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking socat (1.7.3.3-2) ... 2021-03-09T08:54:10.759 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../075-python3-json-pointer_2.0-0ubuntu1_all.deb ... 2021-03-09T08:54:10.803 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-json-pointer (2.0-0ubuntu1) ... 2021-03-09T08:54:11.064 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-03-09T08:54:11.079 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../082-libxslt1.1_1.1.34-4_amd64.deb ... 2021-03-09T08:54:11.114 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2021-03-09T08:54:11.157 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jsonpatch. 2021-03-09T08:54:11.173 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../076-python3-jsonpatch_1.23-3_all.deb ... 2021-03-09T08:54:11.214 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jsonpatch (1.23-3) ... 2021-03-09T08:54:11.392 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package xmlstarlet. 2021-03-09T08:54:11.409 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../083-xmlstarlet_1.6.1-2build1_amd64.deb ... 2021-03-09T08:54:11.450 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2021-03-09T08:54:11.526 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-03-09T08:54:11.542 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../077-ceph-mgr-rook_17.0.0-1742-g89fb622b-1focal_all.deb ... 2021-03-09T08:54:11.584 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-rook (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:11.779 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-test. 2021-03-09T08:54:11.793 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../084-ceph-test_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:54:11.837 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-test (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:11.937 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libonig5:amd64. 2021-03-09T08:54:11.951 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../078-libonig5_6.9.4-1_amd64.deb ... 2021-03-09T08:54:12.028 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2021-03-09T08:54:12.382 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjq1:amd64. 2021-03-09T08:54:12.395 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../079-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2021-03-09T08:54:12.440 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2021-03-09T08:54:12.742 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package jq. 2021-03-09T08:54:12.756 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../080-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2021-03-09T08:54:12.792 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2021-03-09T08:54:13.139 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package socat. 2021-03-09T08:54:13.152 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../081-socat_1.7.3.3-2_amd64.deb ... 2021-03-09T08:54:13.189 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking socat (1.7.3.3-2) ... 2021-03-09T08:54:13.576 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-03-09T08:54:13.590 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../082-libxslt1.1_1.1.34-4_amd64.deb ... 2021-03-09T08:54:13.659 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2021-03-09T08:54:13.937 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package xmlstarlet. 2021-03-09T08:54:13.951 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../083-xmlstarlet_1.6.1-2build1_amd64.deb ... 2021-03-09T08:54:13.995 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2021-03-09T08:54:14.358 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-test. 2021-03-09T08:54:14.372 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../084-ceph-test_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:54:14.416 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-test (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:16.116 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libcephfs-dev. 2021-03-09T08:54:16.129 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../085-libcephfs-dev_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:54:16.158 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libcephfs-dev (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:16.544 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package nvme-cli. 2021-03-09T08:54:16.558 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../086-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2021-03-09T08:54:16.620 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2021-03-09T08:54:17.066 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-atomicwrites. 2021-03-09T08:54:17.081 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../087-python3-atomicwrites_1.1.5-2build1_all.deb ... 2021-03-09T08:54:17.124 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2021-03-09T08:54:17.385 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-attr. 2021-03-09T08:54:17.400 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../088-python3-attr_19.3.0-2_all.deb ... 2021-03-09T08:54:17.435 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-attr (19.3.0-2) ... 2021-03-09T08:54:17.763 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-webencodings. 2021-03-09T08:54:17.776 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../089-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2021-03-09T08:54:17.804 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2021-03-09T08:54:18.082 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-html5lib. 2021-03-09T08:54:18.096 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../090-python3-html5lib_1.0.1-2_all.deb ... 2021-03-09T08:54:18.140 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2021-03-09T08:54:18.477 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-03-09T08:54:18.490 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../091-python3-lxml_4.5.0-1ubuntu0.2_amd64.deb ... 2021-03-09T08:54:18.551 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.2) ... 2021-03-09T08:54:18.753 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libcephfs-dev. 2021-03-09T08:54:18.769 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../085-libcephfs-dev_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:54:18.811 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libcephfs-dev (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:19.021 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-packaging. 2021-03-09T08:54:19.036 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../092-python3-packaging_20.3-1_all.deb ... 2021-03-09T08:54:19.079 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-packaging (20.3-1) ... 2021-03-09T08:54:19.232 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package nvme-cli. 2021-03-09T08:54:19.246 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../086-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2021-03-09T08:54:19.307 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2021-03-09T08:54:19.432 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pastescript. 2021-03-09T08:54:19.447 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../093-python3-pastescript_2.0.2-4_all.deb ... 2021-03-09T08:54:19.490 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2021-03-09T08:54:19.687 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-atomicwrites. 2021-03-09T08:54:19.704 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../087-python3-atomicwrites_1.1.5-2build1_all.deb ... 2021-03-09T08:54:19.745 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2021-03-09T08:54:19.810 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pluggy. 2021-03-09T08:54:19.843 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../094-python3-pluggy_0.13.0-2_all.deb ... 2021-03-09T08:54:19.869 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2021-03-09T08:54:20.040 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-attr. 2021-03-09T08:54:20.055 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../088-python3-attr_19.3.0-2_all.deb ... 2021-03-09T08:54:20.098 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-attr (19.3.0-2) ... 2021-03-09T08:54:20.155 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-psutil. 2021-03-09T08:54:20.171 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../095-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2021-03-09T08:54:20.213 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2021-03-09T08:54:20.443 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-webencodings. 2021-03-09T08:54:20.456 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../089-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2021-03-09T08:54:20.501 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2021-03-09T08:54:20.557 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-py. 2021-03-09T08:54:20.572 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../096-python3-py_1.8.1-1_all.deb ... 2021-03-09T08:54:20.624 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-py (1.8.1-1) ... 2021-03-09T08:54:20.803 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-html5lib. 2021-03-09T08:54:20.818 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../090-python3-html5lib_1.0.1-2_all.deb ... 2021-03-09T08:54:20.853 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2021-03-09T08:54:20.952 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyinotify. 2021-03-09T08:54:20.966 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../097-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2021-03-09T08:54:21.010 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2021-03-09T08:54:21.148 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-03-09T08:54:21.165 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../091-python3-lxml_4.5.0-1ubuntu0.2_amd64.deb ... 2021-03-09T08:54:21.198 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.2) ... 2021-03-09T08:54:21.330 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-wcwidth. 2021-03-09T08:54:21.344 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../098-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2021-03-09T08:54:21.388 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2021-03-09T08:54:21.627 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-packaging. 2021-03-09T08:54:21.642 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../092-python3-packaging_20.3-1_all.deb ... 2021-03-09T08:54:21.685 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-packaging (20.3-1) ... 2021-03-09T08:54:21.690 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pytest. 2021-03-09T08:54:21.707 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2021-03-09T08:54:21.749 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pytest (4.6.9-1) ... 2021-03-09T08:54:22.021 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pastescript. 2021-03-09T08:54:22.036 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../093-python3-pastescript_2.0.2-4_all.deb ... 2021-03-09T08:54:22.079 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2021-03-09T08:54:22.111 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-simplejson. 2021-03-09T08:54:22.125 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2021-03-09T08:54:22.160 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2021-03-09T08:54:22.450 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pluggy. 2021-03-09T08:54:22.464 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../094-python3-pluggy_0.13.0-2_all.deb ... 2021-03-09T08:54:22.508 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2021-03-09T08:54:22.539 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package radosgw. 2021-03-09T08:54:22.555 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../101-radosgw_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:54:22.597 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking radosgw (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:22.794 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-psutil. 2021-03-09T08:54:22.809 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../095-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2021-03-09T08:54:22.844 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2021-03-09T08:54:23.164 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-py. 2021-03-09T08:54:23.180 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../096-python3-py_1.8.1-1_all.deb ... 2021-03-09T08:54:23.214 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-py (1.8.1-1) ... 2021-03-09T08:54:23.534 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyinotify. 2021-03-09T08:54:23.547 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../097-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2021-03-09T08:54:23.591 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2021-03-09T08:54:23.859 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package rbd-fuse. 2021-03-09T08:54:23.872 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../102-rbd-fuse_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:54:23.908 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking rbd-fuse (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:23.984 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-wcwidth. 2021-03-09T08:54:24.001 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../098-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2021-03-09T08:54:24.198 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2021-03-09T08:54:24.209 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-attr (19.3.0-2) ... 2021-03-09T08:54:24.481 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pytest. 2021-03-09T08:54:24.499 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2021-03-09T08:54:24.540 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2021-03-09T08:54:24.572 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pytest (4.6.9-1) ... 2021-03-09T08:54:24.689 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2021-03-09T08:54:24.814 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2021-03-09T08:54:24.893 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-simplejson. 2021-03-09T08:54:24.909 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2021-03-09T08:54:24.950 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2021-03-09T08:54:25.016 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up cephadm (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:25.425 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package radosgw. 2021-03-09T08:54:25.436 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../101-radosgw_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:54:25.480 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking radosgw (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:25.652 INFO:teuthology.orchestra.run.smithi146.stdout:Adding system user cephadm....done 2021-03-09T08:54:25.814 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-waitress (1.4.1-1) ... 2021-03-09T08:54:26.173 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-repoze.lru (0.7-2) ... 2021-03-09T08:54:26.467 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liboath0 (2.6.1-1.3) ... 2021-03-09T08:54:26.576 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-py (1.8.1-1) ... 2021-03-09T08:54:26.749 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package rbd-fuse. 2021-03-09T08:54:26.765 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../102-rbd-fuse_17.0.0-1742-g89fb622b-1focal_amd64.deb ... 2021-03-09T08:54:26.799 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking rbd-fuse (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:26.869 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-joblib (0.14.0-3) ... 2021-03-09T08:54:27.071 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-attr (19.3.0-2) ... 2021-03-09T08:54:27.262 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2021-03-09T08:54:27.395 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2021-03-09T08:54:27.530 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2021-03-09T08:54:27.547 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cachetools (4.0.0-1) ... 2021-03-09T08:54:27.656 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2021-03-09T08:54:27.831 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2021-03-09T08:54:27.857 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up cephadm (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:28.107 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2021-03-09T08:54:28.400 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-ceph-argparse (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:28.477 INFO:teuthology.orchestra.run.smithi005.stdout:Adding system user cephadm....done 2021-03-09T08:54:28.638 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-waitress (1.4.1-1) ... 2021-03-09T08:54:28.684 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2021-03-09T08:54:28.785 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2021-03-09T08:54:28.990 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-repoze.lru (0.7-2) ... 2021-03-09T08:54:29.061 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2021-03-09T08:54:29.258 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liboath0 (2.6.1-1.3) ... 2021-03-09T08:54:29.321 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2021-03-09T08:54:29.367 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-py (1.8.1-1) ... 2021-03-09T08:54:29.697 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2021-03-09T08:54:29.868 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-joblib (0.14.0-3) ... 2021-03-09T08:54:30.014 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2021-03-09T08:54:30.278 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2021-03-09T08:54:30.366 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2021-03-09T08:54:30.580 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cachetools (4.0.0-1) ... 2021-03-09T08:54:30.667 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libjaeger (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:30.776 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2021-03-09T08:54:30.864 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2021-03-09T08:54:31.077 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-tempita (0.5.2-6) ... 2021-03-09T08:54:31.174 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2021-03-09T08:54:31.337 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-packaging (20.3-1) ... 2021-03-09T08:54:31.458 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-ceph-argparse (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:31.613 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2021-03-09T08:54:31.738 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2021-03-09T08:54:31.751 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2021-03-09T08:54:31.877 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2021-03-09T08:54:32.023 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2021-03-09T08:54:32.195 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2021-03-09T08:54:32.410 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2021-03-09T08:54:32.479 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2021-03-09T08:54:32.753 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2021-03-09T08:54:32.864 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2021-03-09T08:54:33.088 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up socat (1.7.3.3-2) ... 2021-03-09T08:54:33.148 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2021-03-09T08:54:33.231 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-ceph-common (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:33.491 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2021-03-09T08:54:33.532 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-html5lib (1.0.1-2) ... 2021-03-09T08:54:33.767 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjaeger (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:33.866 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2021-03-09T08:54:33.902 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2021-03-09T08:54:34.001 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pluggy (0.13.0-2) ... 2021-03-09T08:54:34.228 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-tempita (0.5.2-6) ... 2021-03-09T08:54:34.302 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-json-pointer (2.0-0ubuntu1) ... 2021-03-09T08:54:34.581 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-packaging (20.3-1) ... 2021-03-09T08:54:34.789 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2021-03-09T08:54:34.887 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2021-03-09T08:54:34.916 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2021-03-09T08:54:35.038 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2021-03-09T08:54:35.041 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2021-03-09T08:54:35.309 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2021-03-09T08:54:35.356 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2021-03-09T08:54:35.641 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-logutils (0.3.3-7) ... 2021-03-09T08:54:35.694 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2021-03-09T08:54:35.925 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2021-03-09T08:54:36.059 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2021-03-09T08:54:36.062 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2021-03-09T08:54:36.319 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-prettytable (0.7.2-5) ... 2021-03-09T08:54:36.431 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up socat (1.7.3.3-2) ... 2021-03-09T08:54:36.582 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-ceph-common (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:36.586 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2021-03-09T08:54:36.862 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2021-03-09T08:54:36.884 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-html5lib (1.0.1-2) ... 2021-03-09T08:54:37.211 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-03-09T08:54:37.227 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2021-03-09T08:54:37.361 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pluggy (0.13.0-2) ... 2021-03-09T08:54:37.465 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2021-03-09T08:54:37.608 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2021-03-09T08:54:37.656 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-json-pointer (2.0-0ubuntu1) ... 2021-03-09T08:54:37.922 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2021-03-09T08:54:37.925 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2021-03-09T08:54:38.056 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2021-03-09T08:54:38.208 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2021-03-09T08:54:38.268 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libjs-underscore (1.9.1~dfsg-1) ... 2021-03-09T08:54:38.369 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2021-03-09T08:54:38.495 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-webob (1:1.8.5-2) ... 2021-03-09T08:54:38.543 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2021-03-09T08:54:38.796 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2021-03-09T08:54:38.810 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-logutils (0.3.3-7) ... 2021-03-09T08:54:39.087 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2021-03-09T08:54:39.089 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librados2 (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:39.212 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2021-03-09T08:54:39.232 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2021-03-09T08:54:39.522 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-prettytable (0.7.2-5) ... 2021-03-09T08:54:39.815 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2021-03-09T08:54:40.124 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2021-03-09T08:54:40.500 INFO:teuthology.orchestra.run.smithi005.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-03-09T08:54:40.704 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librgw2 (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:40.777 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2021-03-09T08:54:40.830 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2021-03-09T08:54:40.928 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2021-03-09T08:54:40.956 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pytest (4.6.9-1) ... 2021-03-09T08:54:41.246 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2021-03-09T08:54:41.349 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2021-03-09T08:54:41.605 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjs-underscore (1.9.1~dfsg-1) ... 2021-03-09T08:54:41.642 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jsonpatch (1.23-3) ... 2021-03-09T08:54:41.732 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2021-03-09T08:54:41.740 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/jsonpatch-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-03-09T08:54:41.858 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-webob (1:1.8.5-2) ... 2021-03-09T08:54:41.960 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2021-03-09T08:54:42.220 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-bs4 (4.8.2-1) ... 2021-03-09T08:54:42.242 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2021-03-09T08:54:42.502 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librados2 (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:42.512 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libcephfs2 (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:42.630 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2021-03-09T08:54:42.661 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2021-03-09T08:54:42.965 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libradosstriper1 (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:43.091 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2021-03-09T08:54:43.418 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2021-03-09T08:54:44.159 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librgw2 (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:44.310 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2021-03-09T08:54:44.427 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pytest (4.6.9-1) ... 2021-03-09T08:54:44.480 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librbd1 (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:44.606 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2021-03-09T08:54:44.731 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2021-03-09T08:54:44.804 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2021-03-09T08:54:44.858 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.2) ... 2021-03-09T08:54:45.113 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jsonpatch (1.23-3) ... 2021-03-09T08:54:45.184 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2021-03-09T08:54:45.212 INFO:teuthology.orchestra.run.smithi005.stdout:update-alternatives: using /usr/bin/jsonpatch-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-03-09T08:54:45.327 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rsa (4.0-3) ... 2021-03-09T08:54:45.449 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2021-03-09T08:54:45.603 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-webtest (2.0.32-2) ... 2021-03-09T08:54:45.750 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-bs4 (4.8.2-1) ... 2021-03-09T08:54:45.879 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-modules-core (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:45.980 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-fuse (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:46.101 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libcephfs2 (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:46.200 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-09T08:54:46.201 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-09T08:54:46.218 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2021-03-09T08:54:46.538 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libradosstriper1 (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:46.639 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libcephfs-dev (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:46.655 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2021-03-09T08:54:46.749 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2021-03-09T08:54:47.066 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2021-03-09T08:54:47.066 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pastescript (2.0.2-4) ... 2021-03-09T08:54:47.376 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2021-03-09T08:54:47.720 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rados (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:47.846 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rbd (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:47.997 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up rbd-fuse (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:48.153 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librbd1 (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:48.191 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-routes (2.4.1-2) ... 2021-03-09T08:54:48.254 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2021-03-09T08:54:48.355 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2021-03-09T08:54:48.475 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-google-auth (1.5.1-2) ... 2021-03-09T08:54:48.480 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.2) ... 2021-03-09T08:54:48.777 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rgw (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:48.832 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2021-03-09T08:54:48.903 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cephfs (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:48.958 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rsa (4.0-3) ... 2021-03-09T08:54:49.029 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2021-03-09T08:54:49.259 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-webtest (2.0.32-2) ... 2021-03-09T08:54:49.560 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-modules-core (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:49.678 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-fuse (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:49.898 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-09T08:54:49.899 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-09T08:54:49.940 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-common (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:50.329 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libcephfs-dev (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:50.430 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2021-03-09T08:54:50.468 INFO:teuthology.orchestra.run.smithi146.stdout:Adding group ceph....done 2021-03-09T08:54:50.757 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pastescript (2.0.2-4) ... 2021-03-09T08:54:50.845 INFO:teuthology.orchestra.run.smithi146.stdout:Adding system user ceph....done 2021-03-09T08:54:51.029 INFO:teuthology.orchestra.run.smithi146.stdout:Setting system user ceph properties....done 2021-03-09T08:54:51.058 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2021-03-09T08:54:51.188 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-03-09T08:54:51.419 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rados (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:51.497 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-03-09T08:54:51.545 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rbd (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:51.787 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up rbd-fuse (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:51.950 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-test (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:52.015 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-routes (2.4.1-2) ... 2021-03-09T08:54:52.068 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up radosgw (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:52.316 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-google-auth (1.5.1-2) ... 2021-03-09T08:54:52.344 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-09T08:54:52.344 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-09T08:54:52.667 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rgw (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:52.793 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cephfs (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:52.944 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2021-03-09T08:54:52.986 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-base (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:53.348 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-03-09T08:54:53.796 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mds (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:53.898 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-common (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:54.015 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-09T08:54:54.015 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-09T08:54:54.442 INFO:teuthology.orchestra.run.smithi005.stdout:Adding group ceph....done 2021-03-09T08:54:54.464 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:54.688 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-09T08:54:54.689 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-09T08:54:54.827 INFO:teuthology.orchestra.run.smithi005.stdout:Adding system user ceph....done 2021-03-09T08:54:55.132 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-rook (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:55.295 INFO:teuthology.orchestra.run.smithi005.stdout:Setting system user ceph properties....done 2021-03-09T08:54:55.303 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-osd (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:55.458 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-03-09T08:54:55.773 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-03-09T08:54:56.008 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-09T08:54:56.008 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-09T08:54:56.207 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-test (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:56.333 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up radosgw (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:56.469 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-k8sevents (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:56.607 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-09T08:54:56.607 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-09T08:54:56.636 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:56.779 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mon (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:57.001 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-09T08:54:57.002 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-09T08:54:57.227 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-base (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:57.455 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-cephadm (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:57.590 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-03-09T08:54:57.598 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:57.724 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-dashboard (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:57.868 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for systemd (245.4-4ubuntu3.4) ... 2021-03-09T08:54:58.020 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mds (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:58.160 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for man-db (2.9.1-1) ... 2021-03-09T08:54:58.241 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-09T08:54:58.241 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-09T08:54:58.672 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:58.888 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-09T08:54:58.889 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-09T08:54:59.306 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-rook (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:59.466 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-osd (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:54:59.986 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-09T08:54:59.987 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-09T08:55:00.418 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-k8sevents (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:55:00.578 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:55:00.721 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mon (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:55:00.950 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-09T08:55:00.951 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-09T08:55:01.246 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2021-03-09T08:55:01.388 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-cephadm (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:55:01.540 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:55:01.650 INFO:teuthology.orchestra.run.smithi146.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-09T08:55:01.653 DEBUG:teuthology.parallel:result is None 2021-03-09T08:55:01.691 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-dashboard (17.0.0-1742-g89fb622b-1focal) ... 2021-03-09T08:55:01.835 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for systemd (245.4-4ubuntu3.4) ... 2021-03-09T08:55:02.119 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for man-db (2.9.1-1) ... 2021-03-09T08:55:05.330 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2021-03-09T08:55:05.800 INFO:teuthology.orchestra.run.smithi005.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-09T08:55:05.802 DEBUG:teuthology.parallel:result is None 2021-03-09T08:55:05.803 DEBUG:teuthology.orchestra.run.smithi005:> sudo lsb_release -is 2021-03-09T08:55:05.844 INFO:teuthology.orchestra.run.smithi005.stdout:Ubuntu 2021-03-09T08:55:05.849 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T08:55:05.850 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:55:06.002 DEBUG:teuthology.orchestra.run.smithi005:> dpkg-query -W -f '${Version}' ceph 2021-03-09T08:55:06.027 INFO:teuthology.orchestra.run.smithi005.stdout:17.0.0-1742-g89fb622b-1focal 2021-03-09T08:55:06.028 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-1742-g89fb622b-1focal 2021-03-09T08:55:06.028 INFO:teuthology.task.install:The correct ceph version 17.0.0-1742-g89fb622b-1focal is installed. 2021-03-09T08:55:06.029 DEBUG:teuthology.orchestra.run.smithi146:> sudo lsb_release -is 2021-03-09T08:55:06.079 INFO:teuthology.orchestra.run.smithi146.stdout:Ubuntu 2021-03-09T08:55:06.085 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T08:55:06.086 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:55:06.232 DEBUG:teuthology.orchestra.run.smithi146:> dpkg-query -W -f '${Version}' ceph 2021-03-09T08:55:06.255 INFO:teuthology.orchestra.run.smithi146.stdout:17.0.0-1742-g89fb622b-1focal 2021-03-09T08:55:06.256 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-1742-g89fb622b-1focal 2021-03-09T08:55:06.256 INFO:teuthology.task.install:The correct ceph version 17.0.0-1742-g89fb622b-1focal is installed. 2021-03-09T08:55:06.256 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-03-09T08:55:06.277 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2021-03-09T08:55:06.278 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-09T08:55:06.295 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2021-03-09T08:55:06.296 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-09T08:55:06.311 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-03-09T08:55:06.313 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2021-03-09T08:55:06.313 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/daemon-helper 2021-03-09T08:55:06.352 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-09T08:55:06.362 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2021-03-09T08:55:06.363 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2021-03-09T08:55:06.379 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-09T08:55:06.393 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-03-09T08:55:06.394 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2021-03-09T08:55:06.395 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-09T08:55:06.413 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-09T08:55:06.472 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2021-03-09T08:55:06.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-09T08:55:06.489 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-09T08:55:06.501 INFO:teuthology.run_tasks:Running task cephadm... 2021-03-09T08:55:06.590 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd shutdown pgref assert': True}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4', 'cephadm_mode': 'root'} 2021-03-09T08:55:06.591 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:55:06.592 INFO:tasks.cephadm:Cluster fsid is 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:55:06.593 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2021-03-09T08:55:06.593 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.5', 'mon.c': '[v2:172.21.15.5:3301,v1:172.21.15.5:6790]', 'mon.b': '172.21.15.146'} 2021-03-09T08:55:06.593 INFO:tasks.cephadm:First mon is mon.a on smithi005 2021-03-09T08:55:06.594 INFO:tasks.cephadm:First mgr is y 2021-03-09T08:55:06.594 INFO:tasks.cephadm:Normalizing hostnames... 2021-03-09T08:55:06.597 DEBUG:teuthology.orchestra.run.smithi005:> sudo hostname $(hostname -s) 2021-03-09T08:55:06.615 DEBUG:teuthology.orchestra.run.smithi146:> sudo hostname $(hostname -s) 2021-03-09T08:55:06.633 INFO:tasks.cephadm:Downloading cephadm (repo git://git.ceph.com/ceph-ci.git ref 89fb622b10ca2e58fe0c913a8956377a886e2ab4)... 2021-03-09T08:55:06.633 DEBUG:teuthology.orchestra.run.smithi005:> git archive --remote=git://git.ceph.com/ceph-ci.git 89fb622b10ca2e58fe0c913a8956377a886e2ab4 src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2021-03-09T08:55:06.771 DEBUG:teuthology.orchestra.run.smithi146:> git archive --remote=git://git.ceph.com/ceph-ci.git 89fb622b10ca2e58fe0c913a8956377a886e2ab4 src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2021-03-09T08:55:06.910 DEBUG:teuthology.orchestra.run.smithi005:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-03-09T08:55:06.920 DEBUG:teuthology.orchestra.run.smithi146:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-03-09T08:55:06.963 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph 2021-03-09T08:55:06.981 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph 2021-03-09T08:55:06.999 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 777 /etc/ceph 2021-03-09T08:55:07.014 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /etc/ceph 2021-03-09T08:55:07.028 INFO:tasks.cephadm:Writing seed config... 2021-03-09T08:55:07.031 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2021-03-09T08:55:07.031 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2021-03-09T08:55:07.031 INFO:tasks.cephadm: override: [client] debug ms = 1 2021-03-09T08:55:07.032 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2021-03-09T08:55:07.032 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2021-03-09T08:55:07.032 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2021-03-09T08:55:07.032 INFO:tasks.cephadm: override: [global] ms type = async 2021-03-09T08:55:07.033 INFO:tasks.cephadm: override: [mon] debug mon = 20 2021-03-09T08:55:07.033 INFO:tasks.cephadm: override: [mon] debug ms = 1 2021-03-09T08:55:07.033 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2021-03-09T08:55:07.034 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2021-03-09T08:55:07.034 INFO:tasks.cephadm: override: [osd] debug filestore = 20 2021-03-09T08:55:07.034 INFO:tasks.cephadm: override: [osd] debug journal = 20 2021-03-09T08:55:07.035 INFO:tasks.cephadm: override: [osd] debug ms = 20 2021-03-09T08:55:07.035 INFO:tasks.cephadm: override: [osd] debug osd = 25 2021-03-09T08:55:07.035 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2021-03-09T08:55:07.036 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2021-03-09T08:55:07.036 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2021-03-09T08:55:07.063 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 252abbe0-80b5-11eb-907b-001a4aab830c mon election default strategy = 3 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug filestore = 20 debug journal = 20 debug ms = 20 debug osd = 25 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2021-03-09T08:55:07.064 DEBUG:teuthology.orchestra.run.smithi005:mon.a> sudo journalctl -f -n 0 -u ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service 2021-03-09T08:55:07.066 DEBUG:teuthology.orchestra.run.smithi005:mgr.y> sudo journalctl -f -n 0 -u ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mgr.y.service 2021-03-09T08:55:07.069 INFO:tasks.cephadm:Bootstrapping... 2021-03-09T08:55:07.069 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -v bootstrap --fsid 252abbe0-80b5-11eb-907b-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.5 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2021-03-09T08:55:07.105 INFO:journalctl@ceph.mon.a.smithi005.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-09T08:55:07.110 INFO:journalctl@ceph.mgr.y.smithi005.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-09T08:55:07.352 INFO:teuthology.orchestra.run.smithi005.stderr:Could not locate podman: podman not found 2021-03-09T08:55:07.352 INFO:teuthology.orchestra.run.smithi005.stderr:Verifying podman|docker is present... 2021-03-09T08:55:07.353 INFO:teuthology.orchestra.run.smithi005.stderr:Verifying lvm2 is present... 2021-03-09T08:55:07.353 INFO:teuthology.orchestra.run.smithi005.stderr:Verifying time synchronization is in place... 2021-03-09T08:55:07.353 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-enabled chrony.service 2021-03-09T08:55:07.360 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-03-09T08:55:07.361 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-active chrony.service 2021-03-09T08:55:07.368 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: inactive 2021-03-09T08:55:07.369 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-enabled chronyd.service 2021-03-09T08:55:07.375 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2021-03-09T08:55:07.376 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-active chronyd.service 2021-03-09T08:55:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: inactive 2021-03-09T08:55:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-enabled systemd-timesyncd.service 2021-03-09T08:55:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: masked 2021-03-09T08:55:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-active systemd-timesyncd.service 2021-03-09T08:55:07.394 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: inactive 2021-03-09T08:55:07.394 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-enabled ntpd.service 2021-03-09T08:55:07.400 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2021-03-09T08:55:07.401 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-active ntpd.service 2021-03-09T08:55:07.406 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: inactive 2021-03-09T08:55:07.407 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-enabled ntp.service 2021-03-09T08:55:07.413 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: enabled 2021-03-09T08:55:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-active ntp.service 2021-03-09T08:55:07.420 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: active 2021-03-09T08:55:07.421 INFO:teuthology.orchestra.run.smithi005.stderr:Unit ntp.service is enabled and running 2021-03-09T08:55:07.421 INFO:teuthology.orchestra.run.smithi005.stderr:Repeating the final host check... 2021-03-09T08:55:07.421 INFO:teuthology.orchestra.run.smithi005.stderr:podman|docker (/usr/bin/docker) is present 2021-03-09T08:55:07.422 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl is present 2021-03-09T08:55:07.422 INFO:teuthology.orchestra.run.smithi005.stderr:lvcreate is present 2021-03-09T08:55:07.422 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-enabled chrony.service 2021-03-09T08:55:07.427 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-03-09T08:55:07.428 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-active chrony.service 2021-03-09T08:55:07.434 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: inactive 2021-03-09T08:55:07.435 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-enabled chronyd.service 2021-03-09T08:55:07.440 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2021-03-09T08:55:07.441 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-active chronyd.service 2021-03-09T08:55:07.448 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: inactive 2021-03-09T08:55:07.449 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-enabled systemd-timesyncd.service 2021-03-09T08:55:07.456 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: masked 2021-03-09T08:55:07.457 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-active systemd-timesyncd.service 2021-03-09T08:55:07.463 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: inactive 2021-03-09T08:55:07.464 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-enabled ntpd.service 2021-03-09T08:55:07.469 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2021-03-09T08:55:07.471 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-active ntpd.service 2021-03-09T08:55:07.475 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: inactive 2021-03-09T08:55:07.476 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-enabled ntp.service 2021-03-09T08:55:07.484 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: enabled 2021-03-09T08:55:07.484 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl is-active ntp.service 2021-03-09T08:55:07.491 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: active 2021-03-09T08:55:07.493 INFO:teuthology.orchestra.run.smithi005.stderr:Unit ntp.service is enabled and running 2021-03-09T08:55:07.494 INFO:teuthology.orchestra.run.smithi005.stderr:Host looks OK 2021-03-09T08:55:07.494 INFO:teuthology.orchestra.run.smithi005.stderr:Cluster fsid: 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:55:07.494 INFO:teuthology.orchestra.run.smithi005.stderr:Acquiring lock 139653478086016 on /run/cephadm/252abbe0-80b5-11eb-907b-001a4aab830c.lock 2021-03-09T08:55:07.495 INFO:teuthology.orchestra.run.smithi005.stderr:Lock 139653478086016 acquired on /run/cephadm/252abbe0-80b5-11eb-907b-001a4aab830c.lock 2021-03-09T08:55:07.495 INFO:teuthology.orchestra.run.smithi005.stderr:Verifying IP 172.21.15.5 port 3300 ... 2021-03-09T08:55:07.496 INFO:teuthology.orchestra.run.smithi005.stderr:Verifying IP 172.21.15.5 port 6789 ... 2021-03-09T08:55:07.496 INFO:teuthology.orchestra.run.smithi005.stderr:Base mon IP is 172.21.15.5, final addrv is [v2:172.21.15.5:3300,v1:172.21.15.5:6789] 2021-03-09T08:55:07.496 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/sbin/ip route ls 2021-03-09T08:55:07.497 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/sbin/ip: default via 172.21.15.254 dev ens1f0 2021-03-09T08:55:07.497 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/sbin/ip: 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2021-03-09T08:55:07.498 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.5 2021-03-09T08:55:07.498 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/sbin/ip -6 route ls 2021-03-09T08:55:07.501 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2021-03-09T08:55:07.502 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2021-03-09T08:55:07.502 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1789sec hoplimit 64 pref medium 2021-03-09T08:55:07.502 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/sbin/ip -6 addr ls 2021-03-09T08:55:07.506 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2021-03-09T08:55:07.506 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/sbin/ip: inet6 ::1/128 scope host 2021-03-09T08:55:07.507 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/sbin/ip: valid_lft forever preferred_lft forever 2021-03-09T08:55:07.507 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2021-03-09T08:55:07.507 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/sbin/ip: inet6 fe80::ec4:7aff:febd:1442/64 scope link 2021-03-09T08:55:07.507 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/sbin/ip: valid_lft forever preferred_lft forever 2021-03-09T08:55:07.508 INFO:teuthology.orchestra.run.smithi005.stderr:Mon IP 172.21.15.5 is in CIDR network 172.21.0.0/20 2021-03-09T08:55:07.508 INFO:teuthology.orchestra.run.smithi005.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2021-03-09T08:55:07.509 INFO:teuthology.orchestra.run.smithi005.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4... 2021-03-09T08:55:07.509 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker pull quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:55:08.492 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/docker: 89fb622b10ca2e58fe0c913a8956377a886e2ab4: Pulling from ceph-ci/ceph 2021-03-09T08:55:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/docker: 889524391d55: Pulling fs layer 2021-03-09T08:55:08.494 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/docker: 1f4535046904: Pulling fs layer 2021-03-09T08:55:09.219 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/docker: 889524391d55: Download complete 2021-03-09T08:55:11.269 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/docker: 1f4535046904: Verifying Checksum 2021-03-09T08:55:11.270 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/docker: 1f4535046904: Download complete 2021-03-09T08:55:11.846 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/docker: 889524391d55: Pull complete 2021-03-09T08:55:22.185 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/docker: 1f4535046904: Pull complete 2021-03-09T08:55:22.288 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/docker: Digest: sha256:77c44d4078705828ca023503a532ac791e03b17fe2f773bc0aaeb958f6514e41 2021-03-09T08:55:22.344 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/docker: Status: Downloaded newer image for quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:55:22.345 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/docker: quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:55:22.349 INFO:teuthology.orchestra.run.smithi005.stderr:Extracting ceph user uid/gid from container image... 2021-03-09T08:55:22.350 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -c %u %g /var/lib/ceph 2021-03-09T08:55:33.485 INFO:teuthology.orchestra.run.smithi005.stderr:stat: 167 167 2021-03-09T08:55:34.158 INFO:teuthology.orchestra.run.smithi005.stderr:Creating initial keys... 2021-03-09T08:55:34.159 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 --gen-print-key 2021-03-09T08:55:35.210 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-authtool: AQAHOEdgZOeyDBAARxwOkDoXxmKWYPMnQMBWVw== 2021-03-09T08:55:35.799 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 --gen-print-key 2021-03-09T08:55:36.772 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-authtool: AQAIOEdgsEUtLhAAPvnPlnCPFUHFdxsG/pNiCw== 2021-03-09T08:55:37.364 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 --gen-print-key 2021-03-09T08:55:38.347 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-authtool: AQAKOEdgWOXgFBAAkO24jy2QnLgUamYdkfWUOg== 2021-03-09T08:55:38.939 INFO:teuthology.orchestra.run.smithi005.stderr:Creating initial monmap... 2021-03-09T08:55:38.940 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpngoo88d0:/tmp/monmap:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 --create --clobber --fsid 252abbe0-80b5-11eb-907b-001a4aab830c --addv a [v2:172.21.15.5:3300,v1:172.21.15.5:6789] /tmp/monmap 2021-03-09T08:55:39.984 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2021-03-09T08:55:39.985 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2021-03-09T08:55:39.985 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:55:39.985 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-03-09T08:55:40.556 INFO:teuthology.orchestra.run.smithi005.stderr:monmaptool for a [v2:172.21.15.5:3300,v1:172.21.15.5:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2021-03-09T08:55:40.556 INFO:teuthology.orchestra.run.smithi005.stderr:setting min_mon_release = octopus 2021-03-09T08:55:40.557 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/monmaptool: set fsid to 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:55:40.557 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-03-09T08:55:40.557 INFO:teuthology.orchestra.run.smithi005.stderr: 2021-03-09T08:55:40.557 INFO:teuthology.orchestra.run.smithi005.stderr:Creating mon... 2021-03-09T08:55:40.558 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /var/lib/ceph/252abbe0-80b5-11eb-907b-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmpiirocs9p:/tmp/keyring:z -v /tmp/ceph-tmpngoo88d0:/tmp/monmap:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 --mkfs -i a --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2021-03-09T08:55:41.541 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.537+0000 7f2ff5a74700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-09T08:55:41.542 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 1 imported monmap: 2021-03-09T08:55:41.542 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: epoch 0 2021-03-09T08:55:41.542 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:55:41.543 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: last_changed 2021-03-09T08:55:39.986925+0000 2021-03-09T08:55:41.543 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: created 2021-03-09T08:55:39.986925+0000 2021-03-09T08:55:41.544 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: min_mon_release 15 (octopus) 2021-03-09T08:55:41.544 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: election_strategy: 1 2021-03-09T08:55:41.544 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2021-03-09T08:55:41.548 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.548 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 0 /usr/bin/ceph-mon: set fsid to 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:55:41.548 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: RocksDB version: 6.8.1 2021-03-09T08:55:41.548 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.549 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-09T08:55:41.549 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Compile date Mar 9 2021 2021-03-09T08:55:41.549 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: DB SUMMARY 2021-03-09T08:55:41.550 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.550 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2021-03-09T08:55:41.550 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.551 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2021-03-09T08:55:41.551 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.551 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.error_if_exists: 0 2021-03-09T08:55:41.551 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.create_if_missing: 1 2021-03-09T08:55:41.551 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.paranoid_checks: 1 2021-03-09T08:55:41.552 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.env: 0x5651c2d291c0 2021-03-09T08:55:41.552 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.fs: Posix File System 2021-03-09T08:55:41.552 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.info_log: 0x5651cd0feea0 2021-03-09T08:55:41.552 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-09T08:55:41.553 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.statistics: (nil) 2021-03-09T08:55:41.553 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.use_fsync: 0 2021-03-09T08:55:41.553 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.max_log_file_size: 0 2021-03-09T08:55:41.553 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-09T08:55:41.553 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-09T08:55:41.554 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-09T08:55:41.554 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-09T08:55:41.581 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.allow_fallocate: 1 2021-03-09T08:55:41.581 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-09T08:55:41.581 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-09T08:55:41.581 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.use_direct_reads: 0 2021-03-09T08:55:41.582 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-09T08:55:41.582 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-09T08:55:41.582 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.db_log_dir: 2021-03-09T08:55:41.583 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2021-03-09T08:55:41.583 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-09T08:55:41.583 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.max_subcompactions: 1 2021-03-09T08:55:41.583 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.max_background_flushes: -1 2021-03-09T08:55:41.584 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-09T08:55:41.584 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-09T08:55:41.584 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-09T08:55:41.584 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-09T08:55:41.584 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-09T08:55:41.585 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-09T08:55:41.585 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-09T08:55:41.585 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.write_buffer_manager: 0x5651cd0c5230 2021-03-09T08:55:41.585 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-09T08:55:41.585 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-09T08:55:41.586 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-09T08:55:41.586 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-09T08:55:41.586 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-09T08:55:41.586 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-09T08:55:41.586 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-09T08:55:41.587 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-09T08:55:41.587 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-09T08:55:41.587 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.unordered_write: 0 2021-03-09T08:55:41.587 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-09T08:55:41.587 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-09T08:55:41.588 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-09T08:55:41.588 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-09T08:55:41.588 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.row_cache: None 2021-03-09T08:55:41.588 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.wal_filter: None 2021-03-09T08:55:41.588 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-09T08:55:41.589 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-09T08:55:41.589 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.preserve_deletes: 0 2021-03-09T08:55:41.589 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.two_write_queues: 0 2021-03-09T08:55:41.589 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-09T08:55:41.589 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.atomic_flush: 0 2021-03-09T08:55:41.590 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-09T08:55:41.590 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-09T08:55:41.590 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-09T08:55:41.590 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.log_readahead_size: 0 2021-03-09T08:55:41.591 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-09T08:55:41.591 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.max_background_jobs: 2 2021-03-09T08:55:41.591 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.max_background_compactions: -1 2021-03-09T08:55:41.591 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-09T08:55:41.591 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-09T08:55:41.592 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-09T08:55:41.592 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-09T08:55:41.592 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-09T08:55:41.592 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-09T08:55:41.592 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-09T08:55:41.593 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-09T08:55:41.593 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.max_open_files: -1 2021-03-09T08:55:41.593 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-09T08:55:41.593 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-09T08:55:41.593 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-09T08:55:41.594 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-09T08:55:41.594 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Compression algorithms supported: 2021-03-09T08:55:41.594 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-09T08:55:41.594 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: kZSTD supported: 0 2021-03-09T08:55:41.594 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: kXpressCompression supported: 0 2021-03-09T08:55:41.595 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-09T08:55:41.595 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: kLZ4Compression supported: 1 2021-03-09T08:55:41.595 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: kBZip2Compression supported: 0 2021-03-09T08:55:41.595 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: kZlibCompression supported: 1 2021-03-09T08:55:41.596 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: kSnappyCompression supported: 1 2021-03-09T08:55:41.596 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-09T08:55:41.596 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.541+0000 7f2ff5a74700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2021-03-09T08:55:41.596 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.597 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2021-03-09T08:55:41.597 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.597 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-09T08:55:41.597 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.598 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-09T08:55:41.598 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.merge_operator: 2021-03-09T08:55:41.598 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compaction_filter: None 2021-03-09T08:55:41.598 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-09T08:55:41.598 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-09T08:55:41.599 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-09T08:55:41.599 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5651c4e14218) 2021-03-09T08:55:41.599 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2021-03-09T08:55:41.599 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-09T08:55:41.599 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-09T08:55:41.600 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2021-03-09T08:55:41.600 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: index_type: 0 2021-03-09T08:55:41.600 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2021-03-09T08:55:41.600 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: index_shortening: 1 2021-03-09T08:55:41.600 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2021-03-09T08:55:41.601 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2021-03-09T08:55:41.601 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: checksum: 1 2021-03-09T08:55:41.601 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2021-03-09T08:55:41.601 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: block_cache: 0x5651cd04c110 2021-03-09T08:55:41.601 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2021-03-09T08:55:41.602 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: block_cache_options: 2021-03-09T08:55:41.602 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: capacity : 536870912 2021-03-09T08:55:41.602 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2021-03-09T08:55:41.603 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2021-03-09T08:55:41.603 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2021-03-09T08:55:41.603 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2021-03-09T08:55:41.603 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2021-03-09T08:55:41.604 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: block_size: 4096 2021-03-09T08:55:41.604 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2021-03-09T08:55:41.604 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2021-03-09T08:55:41.604 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2021-03-09T08:55:41.604 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2021-03-09T08:55:41.605 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: partition_filters: 0 2021-03-09T08:55:41.605 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2021-03-09T08:55:41.605 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-09T08:55:41.605 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2021-03-09T08:55:41.605 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: verify_compression: 0 2021-03-09T08:55:41.606 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2021-03-09T08:55:41.606 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: format_version: 2 2021-03-09T08:55:41.606 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2021-03-09T08:55:41.606 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: block_align: 0 2021-03-09T08:55:41.606 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.607 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-09T08:55:41.607 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-09T08:55:41.607 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compression: NoCompression 2021-03-09T08:55:41.607 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-09T08:55:41.607 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-09T08:55:41.608 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-09T08:55:41.608 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.num_levels: 7 2021-03-09T08:55:41.608 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-09T08:55:41.608 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-09T08:55:41.609 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-09T08:55:41.609 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-09T08:55:41.609 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-09T08:55:41.609 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-09T08:55:41.609 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-09T08:55:41.610 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-09T08:55:41.610 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-09T08:55:41.610 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-09T08:55:41.610 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-09T08:55:41.610 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-09T08:55:41.611 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-09T08:55:41.611 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-09T08:55:41.611 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-09T08:55:41.611 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-09T08:55:41.611 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-09T08:55:41.612 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-09T08:55:41.612 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-09T08:55:41.612 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-09T08:55:41.612 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-09T08:55:41.612 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-09T08:55:41.613 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-09T08:55:41.613 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-09T08:55:41.613 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-09T08:55:41.613 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-09T08:55:41.613 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-09T08:55:41.614 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-09T08:55:41.614 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-09T08:55:41.614 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-09T08:55:41.615 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-09T08:55:41.615 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-09T08:55:41.615 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-09T08:55:41.615 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-09T08:55:41.615 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-09T08:55:41.616 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-09T08:55:41.616 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-09T08:55:41.616 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-09T08:55:41.616 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-09T08:55:41.616 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-09T08:55:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-09T08:55:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-09T08:55:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-09T08:55:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-09T08:55:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-09T08:55:41.618 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-09T08:55:41.618 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-09T08:55:41.618 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.table_properties_collectors: 2021-03-09T08:55:41.618 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.inplace_update_support: 0 2021-03-09T08:55:41.619 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-09T08:55:41.619 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-09T08:55:41.619 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-09T08:55:41.619 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-09T08:55:41.619 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.bloom_locality: 0 2021-03-09T08:55:41.620 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.max_successive_merges: 0 2021-03-09T08:55:41.620 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-09T08:55:41.620 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-09T08:55:41.620 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-09T08:55:41.620 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-09T08:55:41.621 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.ttl: 2592000 2021-03-09T08:55:41.621 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-09T08:55:41.621 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-09T08:55:41.621 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.621 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.545+0000 7f2ff5a74700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-03-09T08:55:41.622 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.622 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.549+0000 7f2ff5a74700 4 rocksdb: DB pointer 0x5651cd111800 2021-03-09T08:55:41.622 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.549+0000 7f2fe6b05700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-09T08:55:41.622 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.549+0000 7f2fe6b05700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-09T08:55:41.622 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2021-03-09T08:55:41.623 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T08:55:41.623 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-09T08:55:41.623 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-09T08:55:41.623 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T08:55:41.623 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-09T08:55:41.624 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-09T08:55:41.624 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T08:55:41.624 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.624 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-03-09T08:55:41.625 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T08:55:41.625 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T08:55:41.625 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T08:55:41.625 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T08:55:41.626 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.626 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-03-09T08:55:41.626 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T08:55:41.626 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T08:55:41.626 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T08:55:41.627 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T08:55:41.627 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T08:55:41.627 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T08:55:41.627 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T08:55:41.627 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-03-09T08:55:41.628 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T08:55:41.628 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T08:55:41.628 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T08:55:41.628 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.629 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-03-09T08:55:41.629 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.629 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-03-09T08:55:41.629 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T08:55:41.629 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T08:55:41.630 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T08:55:41.630 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T08:55:41.630 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.630 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-03-09T08:55:41.630 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T08:55:41.631 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T08:55:41.631 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T08:55:41.631 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T08:55:41.631 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T08:55:41.631 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T08:55:41.632 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T08:55:41.632 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-03-09T08:55:41.632 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T08:55:41.632 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T08:55:41.632 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T08:55:41.633 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.633 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-03-09T08:55:41.633 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: 2021-03-09T08:55:41.633 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.553+0000 7f2ff5a74700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2021-03-09T08:55:41.633 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.553+0000 7f2ff5a74700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2021-03-09T08:55:41.634 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-mon: debug 2021-03-09T08:55:41.553+0000 7f2ff5a74700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2021-03-09T08:55:42.196 INFO:teuthology.orchestra.run.smithi005.stderr:create mon.a on 2021-03-09T08:55:42.197 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:55:42.204 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl disable ceph.target 2021-03-09T08:55:42.223 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2021-03-09T08:55:42.404 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl enable ceph.target 2021-03-09T08:55:42.410 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2021-03-09T08:55:42.585 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl start ceph.target 2021-03-09T08:55:42.594 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl enable ceph-252abbe0-80b5-11eb-907b-001a4aab830c.target 2021-03-09T08:55:42.600 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-252abbe0-80b5-11eb-907b-001a4aab830c.target -> /etc/systemd/system/ceph-252abbe0-80b5-11eb-907b-001a4aab830c.target. 2021-03-09T08:55:42.600 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-252abbe0-80b5-11eb-907b-001a4aab830c.target -> /etc/systemd/system/ceph-252abbe0-80b5-11eb-907b-001a4aab830c.target. 2021-03-09T08:55:42.769 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl start ceph-252abbe0-80b5-11eb-907b-001a4aab830c.target 2021-03-09T08:55:42.777 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl daemon-reload 2021-03-09T08:55:42.965 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl stop ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a 2021-03-09T08:55:42.977 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl reset-failed ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a 2021-03-09T08:55:42.982 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Failed to reset failed state of unit ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Unit ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service not loaded. 2021-03-09T08:55:42.983 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl enable ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a 2021-03-09T08:55:42.991 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Created symlink /etc/systemd/system/ceph-252abbe0-80b5-11eb-907b-001a4aab830c.target.wants/ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service -> /etc/systemd/system/ceph-252abbe0-80b5-11eb-907b-001a4aab830c@.service. 2021-03-09T08:55:43.156 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl start ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a 2021-03-09T08:55:43.170 INFO:teuthology.orchestra.run.smithi005.stderr:firewalld does not appear to be present 2021-03-09T08:55:43.170 INFO:teuthology.orchestra.run.smithi005.stderr:Not possible to enable service . firewalld.service is not available 2021-03-09T08:55:43.170 INFO:teuthology.orchestra.run.smithi005.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-03-09T08:55:43.171 INFO:teuthology.orchestra.run.smithi005.stderr:Waiting for mon to start... 2021-03-09T08:55:43.172 INFO:teuthology.orchestra.run.smithi005.stderr:Waiting for mon... 2021-03-09T08:55:43.172 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/252abbe0-80b5-11eb-907b-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 status 2021-03-09T08:55:44.702 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.701+0000 7fb7ab402700 1 Processor -- start 2021-03-09T08:55:44.703 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.701+0000 7fb7ab402700 1 -- start start 2021-03-09T08:55:44.703 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.701+0000 7fb7ab402700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a406fc90 0x7fb7a406e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:44.703 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.701+0000 7fb7ab402700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a40700b0 con 0x7fb7a406fc90 2021-03-09T08:55:44.704 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.701+0000 7fb7aa400700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a406fc90 0x7fb7a406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:44.704 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.701+0000 7fb7aa400700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a406fc90 0x7fb7a406e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46226/0 (socket says 172.21.15.5:46226) 2021-03-09T08:55:44.704 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.701+0000 7fb7aa400700 1 -- 172.21.15.5:0/2622154254 learned_addr learned my addr 172.21.15.5:0/2622154254 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:55:44.706 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.701+0000 7fb7aa400700 1 -- 172.21.15.5:0/2622154254 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a406e720 con 0x7fb7a406fc90 2021-03-09T08:55:44.706 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.701+0000 7fb7aa400700 1 --2- 172.21.15.5:0/2622154254 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a406fc90 0x7fb7a406e230 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb79c009cc0 tx=0x7fb79c0092d0).ready entity=mon.0 client_cookie=30b260c35b14b843 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:44.706 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.701+0000 7fb7a93fe700 1 -- 172.21.15.5:0/2622154254 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb79c00bbf0 con 0x7fb7a406fc90 2021-03-09T08:55:44.707 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.701+0000 7fb7a93fe700 1 -- 172.21.15.5:0/2622154254 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79c00bd50 con 0x7fb7a406fc90 2021-03-09T08:55:44.707 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.701+0000 7fb7a93fe700 1 -- 172.21.15.5:0/2622154254 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb79c011400 con 0x7fb7a406fc90 2021-03-09T08:55:44.707 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7ab402700 1 -- 172.21.15.5:0/2622154254 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a406fc90 msgr2=0x7fb7a406e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:55:44.707 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7ab402700 1 --2- 172.21.15.5:0/2622154254 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a406fc90 0x7fb7a406e230 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb79c009cc0 tx=0x7fb79c0092d0).stop 2021-03-09T08:55:44.708 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7ab402700 1 -- 172.21.15.5:0/2622154254 shutdown_connections 2021-03-09T08:55:44.708 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7ab402700 1 --2- 172.21.15.5:0/2622154254 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a406fc90 0x7fb7a406e230 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:44.708 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7ab402700 1 -- 172.21.15.5:0/2622154254 >> 172.21.15.5:0/2622154254 conn(0x7fb7a40f5120 msgr2=0x7fb7a40f7560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:55:44.709 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7ab402700 1 -- 172.21.15.5:0/2622154254 shutdown_connections 2021-03-09T08:55:44.709 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7ab402700 1 -- 172.21.15.5:0/2622154254 wait complete. 2021-03-09T08:55:44.709 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7ab402700 1 Processor -- start 2021-03-09T08:55:44.710 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7ab402700 1 -- start start 2021-03-09T08:55:44.713 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7ab402700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a406fc90 0x7fb7a4109b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:44.714 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7ab402700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a410a050 con 0x7fb7a406fc90 2021-03-09T08:55:44.714 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7aa400700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a406fc90 0x7fb7a4109b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:44.715 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7aa400700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a406fc90 0x7fb7a4109b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46228/0 (socket says 172.21.15.5:46228) 2021-03-09T08:55:44.715 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7aa400700 1 -- 172.21.15.5:0/2387374456 learned_addr learned my addr 172.21.15.5:0/2387374456 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:55:44.716 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7aa400700 1 -- 172.21.15.5:0/2387374456 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a410a190 con 0x7fb7a406fc90 2021-03-09T08:55:44.716 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7aa400700 1 --2- 172.21.15.5:0/2387374456 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a406fc90 0x7fb7a4109b60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fb79c009310 tx=0x7fb79c017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:44.716 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb792ffd700 1 -- 172.21.15.5:0/2387374456 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb79c00b780 con 0x7fb7a406fc90 2021-03-09T08:55:44.717 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb792ffd700 1 -- 172.21.15.5:0/2387374456 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79c011e20 con 0x7fb7a406fc90 2021-03-09T08:55:44.717 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb792ffd700 1 -- 172.21.15.5:0/2387374456 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb79c011400 con 0x7fb7a406fc90 2021-03-09T08:55:44.717 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7ab402700 1 -- 172.21.15.5:0/2387374456 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a410a3e0 con 0x7fb7a406fc90 2021-03-09T08:55:44.718 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.705+0000 7fb7ab402700 1 -- 172.21.15.5:0/2387374456 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a410aed0 con 0x7fb7a406fc90 2021-03-09T08:55:44.718 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.709+0000 7fb792ffd700 1 -- 172.21.15.5:0/2387374456 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7fb79c022030 con 0x7fb7a406fc90 2021-03-09T08:55:44.719 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.709+0000 7fb792ffd700 1 -- 172.21.15.5:0/2387374456 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb79c0259e0 con 0x7fb7a406fc90 2021-03-09T08:55:44.719 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.709+0000 7fb7ab402700 1 -- 172.21.15.5:0/2387374456 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a40589f0 con 0x7fb7a406fc90 2021-03-09T08:55:44.720 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.713+0000 7fb792ffd700 1 -- 172.21.15.5:0/2387374456 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7fb79c02a060 con 0x7fb7a406fc90 2021-03-09T08:55:44.850 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.845+0000 7fb7ab402700 1 -- 172.21.15.5:0/2387374456 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fb7a4045280 con 0x7fb7a406fc90 2021-03-09T08:55:44.850 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.845+0000 7fb792ffd700 1 -- 172.21.15.5:0/2387374456 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+316 (secure 0 0 0) 0x7fb79c020070 con 0x7fb7a406fc90 2021-03-09T08:55:44.852 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: cluster: 2021-03-09T08:55:44.852 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: id: 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:55:44.853 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: health: HEALTH_OK 2021-03-09T08:55:44.853 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.853 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: services: 2021-03-09T08:55:44.853 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.22581s) 2021-03-09T08:55:44.854 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mgr: no daemons active 2021-03-09T08:55:44.854 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2021-03-09T08:55:44.854 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.854 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: data: 2021-03-09T08:55:44.855 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2021-03-09T08:55:44.855 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2021-03-09T08:55:44.855 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2021-03-09T08:55:44.855 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: pgs: 2021-03-09T08:55:44.855 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.856 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.845+0000 7fb7ab402700 1 -- 172.21.15.5:0/2387374456 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a406fc90 msgr2=0x7fb7a4109b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:55:44.856 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.845+0000 7fb7ab402700 1 --2- 172.21.15.5:0/2387374456 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a406fc90 0x7fb7a4109b60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fb79c009310 tx=0x7fb79c017000).stop 2021-03-09T08:55:44.857 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.845+0000 7fb7ab402700 1 -- 172.21.15.5:0/2387374456 shutdown_connections 2021-03-09T08:55:44.857 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.845+0000 7fb7ab402700 1 --2- 172.21.15.5:0/2387374456 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb7a406fc90 0x7fb7a4109b60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:44.857 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.845+0000 7fb7ab402700 1 -- 172.21.15.5:0/2387374456 >> 172.21.15.5:0/2387374456 conn(0x7fb7a40f5120 msgr2=0x7fb7a40fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:55:44.857 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.845+0000 7fb7ab402700 1 -- 172.21.15.5:0/2387374456 shutdown_connections 2021-03-09T08:55:44.857 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:44.845+0000 7fb7ab402700 1 -- 172.21.15.5:0/2387374456 wait complete. 2021-03-09T08:55:44.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:44 smithi005 bash[12361]: cluster 2021-03-09T08:55:44.616442+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2021-03-09T08:55:45.434 INFO:teuthology.orchestra.run.smithi005.stderr:mon is available 2021-03-09T08:55:45.502 INFO:teuthology.orchestra.run.smithi005.stderr:Assimilating anything we can from ceph.conf... 2021-03-09T08:55:45.502 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z -v /var/lib/ceph/252abbe0-80b5-11eb-907b-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 config assimilate-conf -i /var/lib/ceph/mon/ceph-a/config 2021-03-09T08:55:45.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:45 smithi005 bash[12361]: cluster 2021-03-09T08:55:44.622742+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2021-03-09T08:55:45.865 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:45 smithi005 bash[12361]: cluster 2021-03-09T08:55:44.623009+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0]} 2021-03-09T08:55:45.865 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:45 smithi005 bash[12361]: cluster 2021-03-09T08:55:44.626298+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2021-03-09T08:55:45.865 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:45 smithi005 bash[12361]: cluster 2021-03-09T08:55:44.628580+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-03-09T08:55:45.865 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:45 smithi005 bash[12361]: cluster 2021-03-09T08:55:44.629129+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2021-03-09T08:55:45.866 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:45 smithi005 bash[12361]: audit 2021-03-09T08:55:44.848503+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.5:0/2387374456' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2021-03-09T08:55:46.559 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.557+0000 7fda12717700 1 Processor -- start 2021-03-09T08:55:46.559 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.557+0000 7fda12717700 1 -- start start 2021-03-09T08:55:46.560 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.557+0000 7fda12717700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda0c0f9e20 0x7fda0c0fa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:46.561 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.557+0000 7fda12717700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c0fa730 con 0x7fda0c0f9e20 2021-03-09T08:55:46.561 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.557+0000 7fda11715700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda0c0f9e20 0x7fda0c0fa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:46.561 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.557+0000 7fda11715700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda0c0f9e20 0x7fda0c0fa240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46230/0 (socket says 172.21.15.5:46230) 2021-03-09T08:55:46.561 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.557+0000 7fda11715700 1 -- 172.21.15.5:0/124792276 learned_addr learned my addr 172.21.15.5:0/124792276 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:55:46.562 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.557+0000 7fda11715700 1 -- 172.21.15.5:0/124792276 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c0fa870 con 0x7fda0c0f9e20 2021-03-09T08:55:46.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.557+0000 7fda11715700 1 --2- 172.21.15.5:0/124792276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda0c0f9e20 0x7fda0c0fa240 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fda0800b950 tx=0x7fda08006ee0).ready entity=mon.0 client_cookie=7368343aaedac708 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:46.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.561+0000 7fd9fbfff700 1 -- 172.21.15.5:0/124792276 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda08009840 con 0x7fda0c0f9e20 2021-03-09T08:55:46.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.561+0000 7fd9fbfff700 1 -- 172.21.15.5:0/124792276 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda080099a0 con 0x7fda0c0f9e20 2021-03-09T08:55:46.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.561+0000 7fd9fbfff700 1 -- 172.21.15.5:0/124792276 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda08009c50 con 0x7fda0c0f9e20 2021-03-09T08:55:46.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.561+0000 7fda12717700 1 -- 172.21.15.5:0/124792276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda0c0f9e20 msgr2=0x7fda0c0fa240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:55:46.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.561+0000 7fda12717700 1 --2- 172.21.15.5:0/124792276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda0c0f9e20 0x7fda0c0fa240 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fda0800b950 tx=0x7fda08006ee0).stop 2021-03-09T08:55:46.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.561+0000 7fda12717700 1 -- 172.21.15.5:0/124792276 shutdown_connections 2021-03-09T08:55:46.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.561+0000 7fda12717700 1 --2- 172.21.15.5:0/124792276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda0c0f9e20 0x7fda0c0fa240 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:46.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.561+0000 7fda12717700 1 -- 172.21.15.5:0/124792276 >> 172.21.15.5:0/124792276 conn(0x7fda0c0f37a0 msgr2=0x7fda0c0f5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:55:46.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.561+0000 7fda12717700 1 -- 172.21.15.5:0/124792276 shutdown_connections 2021-03-09T08:55:46.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.561+0000 7fda12717700 1 -- 172.21.15.5:0/124792276 wait complete. 2021-03-09T08:55:46.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.565+0000 7fda12717700 1 Processor -- start 2021-03-09T08:55:46.568 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.565+0000 7fda12717700 1 -- start start 2021-03-09T08:55:46.568 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.565+0000 7fda12717700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda0c0f9e20 0x7fda0c105750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:46.569 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.565+0000 7fda12717700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c105c40 con 0x7fda0c0f9e20 2021-03-09T08:55:46.569 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.565+0000 7fda11715700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda0c0f9e20 0x7fda0c105750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:46.569 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.565+0000 7fda11715700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda0c0f9e20 0x7fda0c105750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46232/0 (socket says 172.21.15.5:46232) 2021-03-09T08:55:46.570 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.565+0000 7fda11715700 1 -- 172.21.15.5:0/769611520 learned_addr learned my addr 172.21.15.5:0/769611520 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:55:46.570 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.565+0000 7fda11715700 1 -- 172.21.15.5:0/769611520 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c105d80 con 0x7fda0c0f9e20 2021-03-09T08:55:46.570 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.565+0000 7fda11715700 1 --2- 172.21.15.5:0/769611520 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda0c0f9e20 0x7fda0c105750 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fda08006c70 tx=0x7fda08016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:46.570 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.565+0000 7fd9f9ffb700 1 -- 172.21.15.5:0/769611520 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda08009840 con 0x7fda0c0f9e20 2021-03-09T08:55:46.571 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.565+0000 7fd9f9ffb700 1 -- 172.21.15.5:0/769611520 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0801c5a0 con 0x7fda0c0f9e20 2021-03-09T08:55:46.571 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.565+0000 7fda12717700 1 -- 172.21.15.5:0/769611520 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0c105fd0 con 0x7fda0c0f9e20 2021-03-09T08:55:46.571 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.565+0000 7fd9f9ffb700 1 -- 172.21.15.5:0/769611520 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda0801c870 con 0x7fda0c0f9e20 2021-03-09T08:55:46.571 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.565+0000 7fda12717700 1 -- 172.21.15.5:0/769611520 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0c106b30 con 0x7fda0c0f9e20 2021-03-09T08:55:46.572 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.565+0000 7fd9f9ffb700 1 -- 172.21.15.5:0/769611520 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7fda0800f070 con 0x7fda0c0f9e20 2021-03-09T08:55:46.572 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.569+0000 7fd9f9ffb700 1 -- 172.21.15.5:0/769611520 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fda0801cdd0 con 0x7fda0c0f9e20 2021-03-09T08:55:46.573 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.569+0000 7fda12717700 1 -- 172.21.15.5:0/769611520 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda0c0589f0 con 0x7fda0c0f9e20 2021-03-09T08:55:46.573 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.569+0000 7fd9f9ffb700 1 -- 172.21.15.5:0/769611520 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7fda08016070 con 0x7fda0c0f9e20 2021-03-09T08:55:46.701 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.701+0000 7fda12717700 1 -- 172.21.15.5:0/769611520 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fda0c045280 con 0x7fda0c0f9e20 2021-03-09T08:55:46.706 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.705+0000 7fd9f9ffb700 1 -- 172.21.15.5:0/769611520 <== mon.0 v2:172.21.15.5:3300/0 7 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7fda08006970 con 0x7fda0c0f9e20 2021-03-09T08:55:46.708 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.705+0000 7fd9f9ffb700 1 -- 172.21.15.5:0/769611520 <== mon.0 v2:172.21.15.5:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+336 (secure 0 0 0) 0x7fda0c045280 con 0x7fda0c0f9e20 2021-03-09T08:55:46.709 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.709 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: [global] 2021-03-09T08:55:46.709 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: fsid = 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:55:46.709 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.5:3300,v1:172.21.15.5:6789] 2021-03-09T08:55:46.709 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2021-03-09T08:55:46.710 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2021-03-09T08:55:46.710 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2021-03-09T08:55:46.711 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2021-03-09T08:55:46.711 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.711 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: [mgr] 2021-03-09T08:55:46.711 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2021-03-09T08:55:46.711 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.712 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: [osd] 2021-03-09T08:55:46.712 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2021-03-09T08:55:46.712 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2021-03-09T08:55:46.713 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.709+0000 7fda12717700 1 -- 172.21.15.5:0/769611520 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda0c0f9e20 msgr2=0x7fda0c105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:55:46.714 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.709+0000 7fda12717700 1 --2- 172.21.15.5:0/769611520 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda0c0f9e20 0x7fda0c105750 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fda08006c70 tx=0x7fda08016000).stop 2021-03-09T08:55:46.714 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.709+0000 7fda12717700 1 -- 172.21.15.5:0/769611520 shutdown_connections 2021-03-09T08:55:46.714 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.709+0000 7fda12717700 1 --2- 172.21.15.5:0/769611520 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda0c0f9e20 0x7fda0c105750 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:46.714 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.709+0000 7fda12717700 1 -- 172.21.15.5:0/769611520 >> 172.21.15.5:0/769611520 conn(0x7fda0c0f37a0 msgr2=0x7fda0c0f5640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:55:46.714 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.709+0000 7fda12717700 1 -- 172.21.15.5:0/769611520 shutdown_connections 2021-03-09T08:55:46.716 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:46.709+0000 7fda12717700 1 -- 172.21.15.5:0/769611520 wait complete. 2021-03-09T08:55:47.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:46 smithi005 bash[12361]: audit 2021-03-09T08:55:46.704567+0000 mon.a (mon.0) 8 : audit [INF] from='client.? 172.21.15.5:0/769611520' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-03-09T08:55:47.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:46 smithi005 bash[12361]: audit 2021-03-09T08:55:46.710079+0000 mon.a (mon.0) 9 : audit [INF] from='client.? 172.21.15.5:0/769611520' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2021-03-09T08:55:47.407 INFO:teuthology.orchestra.run.smithi005.stderr:Generating new minimal ceph.conf... 2021-03-09T08:55:47.407 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z -v /var/lib/ceph/252abbe0-80b5-11eb-907b-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-a/config 2021-03-09T08:55:48.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.541+0000 7fcf84c4d700 1 Processor -- start 2021-03-09T08:55:48.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.541+0000 7fcf84c4d700 1 -- start start 2021-03-09T08:55:48.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.541+0000 7fcf84c4d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf8006dbd0 0x7fcf800fea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:48.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.541+0000 7fcf84c4d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800fa3f0 con 0x7fcf8006dbd0 2021-03-09T08:55:48.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.541+0000 7fcf7f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf8006dbd0 0x7fcf800fea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:48.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.541+0000 7fcf7f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf8006dbd0 0x7fcf800fea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46234/0 (socket says 172.21.15.5:46234) 2021-03-09T08:55:48.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.541+0000 7fcf7f7fe700 1 -- 172.21.15.5:0/3831356201 learned_addr learned my addr 172.21.15.5:0/3831356201 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:55:48.567 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.545+0000 7fcf7f7fe700 1 -- 172.21.15.5:0/3831356201 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf800feef0 con 0x7fcf8006dbd0 2021-03-09T08:55:48.567 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.545+0000 7fcf7f7fe700 1 --2- 172.21.15.5:0/3831356201 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf8006dbd0 0x7fcf800fea00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fcf7400b950 tx=0x7fcf74006ee0).ready entity=mon.0 client_cookie=67fa38be195f53d2 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:48.568 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.545+0000 7fcf7e7fc700 1 -- 172.21.15.5:0/3831356201 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf74009840 con 0x7fcf8006dbd0 2021-03-09T08:55:48.568 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.545+0000 7fcf7e7fc700 1 -- 172.21.15.5:0/3831356201 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7fcf74009e00 con 0x7fcf8006dbd0 2021-03-09T08:55:48.568 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.545+0000 7fcf7e7fc700 1 -- 172.21.15.5:0/3831356201 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf74011c80 con 0x7fcf8006dbd0 2021-03-09T08:55:48.569 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.545+0000 7fcf84c4d700 1 -- 172.21.15.5:0/3831356201 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf8006dbd0 msgr2=0x7fcf800fea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:55:48.569 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.545+0000 7fcf84c4d700 1 --2- 172.21.15.5:0/3831356201 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf8006dbd0 0x7fcf800fea00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fcf7400b950 tx=0x7fcf74006ee0).stop 2021-03-09T08:55:48.570 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.545+0000 7fcf84c4d700 1 -- 172.21.15.5:0/3831356201 shutdown_connections 2021-03-09T08:55:48.570 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.545+0000 7fcf84c4d700 1 --2- 172.21.15.5:0/3831356201 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf8006dbd0 0x7fcf800fea00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:48.574 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.545+0000 7fcf84c4d700 1 -- 172.21.15.5:0/3831356201 >> 172.21.15.5:0/3831356201 conn(0x7fcf800f4020 msgr2=0x7fcf800f6440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:55:48.574 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.545+0000 7fcf84c4d700 1 -- 172.21.15.5:0/3831356201 shutdown_connections 2021-03-09T08:55:48.574 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.545+0000 7fcf84c4d700 1 -- 172.21.15.5:0/3831356201 wait complete. 2021-03-09T08:55:48.574 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.549+0000 7fcf84c4d700 1 Processor -- start 2021-03-09T08:55:48.575 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.549+0000 7fcf84c4d700 1 -- start start 2021-03-09T08:55:48.575 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.549+0000 7fcf84c4d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf8006dbd0 0x7fcf8010a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:48.575 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.549+0000 7fcf84c4d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8010a700 con 0x7fcf8006dbd0 2021-03-09T08:55:48.575 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.549+0000 7fcf7f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf8006dbd0 0x7fcf8010a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:48.575 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.549+0000 7fcf7f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf8006dbd0 0x7fcf8010a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46236/0 (socket says 172.21.15.5:46236) 2021-03-09T08:55:48.576 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.549+0000 7fcf7f7fe700 1 -- 172.21.15.5:0/2189108485 learned_addr learned my addr 172.21.15.5:0/2189108485 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:55:48.576 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.549+0000 7fcf7f7fe700 1 -- 172.21.15.5:0/2189108485 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8010a840 con 0x7fcf8006dbd0 2021-03-09T08:55:48.577 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.549+0000 7fcf7f7fe700 1 --2- 172.21.15.5:0/2189108485 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf8006dbd0 0x7fcf8010a210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcf74009420 tx=0x7fcf74006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:48.577 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.549+0000 7fcf67fff700 1 -- 172.21.15.5:0/2189108485 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf74009ca0 con 0x7fcf8006dbd0 2021-03-09T08:55:48.577 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.549+0000 7fcf67fff700 1 -- 172.21.15.5:0/2189108485 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7fcf74011400 con 0x7fcf8006dbd0 2021-03-09T08:55:48.577 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.549+0000 7fcf67fff700 1 -- 172.21.15.5:0/2189108485 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf7401bbf0 con 0x7fcf8006dbd0 2021-03-09T08:55:48.578 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.549+0000 7fcf84c4d700 1 -- 172.21.15.5:0/2189108485 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf8010aa90 con 0x7fcf8006dbd0 2021-03-09T08:55:48.579 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.549+0000 7fcf84c4d700 1 -- 172.21.15.5:0/2189108485 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf8010b5f0 con 0x7fcf8006dbd0 2021-03-09T08:55:48.580 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.549+0000 7fcf67fff700 1 -- 172.21.15.5:0/2189108485 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7fcf7402b070 con 0x7fcf8006dbd0 2021-03-09T08:55:48.580 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.549+0000 7fcf67fff700 1 -- 172.21.15.5:0/2189108485 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fcf74023720 con 0x7fcf8006dbd0 2021-03-09T08:55:48.581 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.553+0000 7fcf84c4d700 1 -- 172.21.15.5:0/2189108485 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf800589f0 con 0x7fcf8006dbd0 2021-03-09T08:55:48.581 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.553+0000 7fcf67fff700 1 -- 172.21.15.5:0/2189108485 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7fcf7401f020 con 0x7fcf8006dbd0 2021-03-09T08:55:48.686 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.685+0000 7fcf84c4d700 1 -- 172.21.15.5:0/2189108485 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fcf800ffc10 con 0x7fcf8006dbd0 2021-03-09T08:55:48.687 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.685+0000 7fcf67fff700 1 -- 172.21.15.5:0/2189108485 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+173 (secure 0 0 0) 0x7fcf74016070 con 0x7fcf8006dbd0 2021-03-09T08:55:48.689 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.685+0000 7fcf84c4d700 1 -- 172.21.15.5:0/2189108485 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf8006dbd0 msgr2=0x7fcf8010a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:55:48.690 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.685+0000 7fcf84c4d700 1 --2- 172.21.15.5:0/2189108485 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf8006dbd0 0x7fcf8010a210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcf74009420 tx=0x7fcf74006ee0).stop 2021-03-09T08:55:48.690 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.685+0000 7fcf84c4d700 1 -- 172.21.15.5:0/2189108485 shutdown_connections 2021-03-09T08:55:48.691 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.685+0000 7fcf84c4d700 1 --2- 172.21.15.5:0/2189108485 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf8006dbd0 0x7fcf8010a210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:48.691 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.685+0000 7fcf84c4d700 1 -- 172.21.15.5:0/2189108485 >> 172.21.15.5:0/2189108485 conn(0x7fcf800f4020 msgr2=0x7fcf800f6370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:55:48.691 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.685+0000 7fcf84c4d700 1 -- 172.21.15.5:0/2189108485 shutdown_connections 2021-03-09T08:55:48.692 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:48.685+0000 7fcf84c4d700 1 -- 172.21.15.5:0/2189108485 wait complete. 2021-03-09T08:55:49.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:48 smithi005 bash[12361]: audit 2021-03-09T08:55:48.689061+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.5:0/2189108485' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:55:49.338 INFO:teuthology.orchestra.run.smithi005.stderr:Restarting the monitor... 2021-03-09T08:55:49.338 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl restart ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a 2021-03-09T08:55:49.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:49 smithi005 systemd[1]: Stopping Ceph mon.a for 252abbe0-80b5-11eb-907b-001a4aab830c... 2021-03-09T08:55:49.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:49 smithi005 bash[12361]: debug 2021-03-09T08:55:49.449+0000 7f6dab9be700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-03-09T08:55:49.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:49 smithi005 bash[12361]: debug 2021-03-09T08:55:49.449+0000 7f6dab9be700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2021-03-09T08:55:50.205 INFO:teuthology.orchestra.run.smithi005.stderr:Setting mon public_network to 172.21.0.0/20 2021-03-09T08:55:50.206 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 config set mon public_network 172.21.0.0/20 2021-03-09T08:55:50.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:50 smithi005 docker[12676]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c-mon.a 2021-03-09T08:55:50.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:50 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Succeeded. 2021-03-09T08:55:50.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:50 smithi005 systemd[1]: Stopped Ceph mon.a for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T08:55:50.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:50 smithi005 systemd[1]: Started Ceph mon.a for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T08:55:51.969 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.965+0000 7f49450fe700 1 Processor -- start 2021-03-09T08:55:51.970 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.965+0000 7f49450fe700 1 -- start start 2021-03-09T08:55:51.970 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.965+0000 7f49450fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49400f9940 0x7f49400f9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:51.971 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.965+0000 7f49450fe700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49400fa250 con 0x7f49400f9940 2021-03-09T08:55:51.971 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.965+0000 7f493ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49400f9940 0x7f49400f9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:51.972 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.965+0000 7f493ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49400f9940 0x7f49400f9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46238/0 (socket says 172.21.15.5:46238) 2021-03-09T08:55:51.972 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.965+0000 7f493ffff700 1 -- 172.21.15.5:0/1740013024 learned_addr learned my addr 172.21.15.5:0/1740013024 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:55:51.972 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.965+0000 7f493ffff700 1 -- 172.21.15.5:0/1740013024 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49400fa390 con 0x7f49400f9940 2021-03-09T08:55:51.973 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.965+0000 7f493ffff700 1 --2- 172.21.15.5:0/1740013024 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49400f9940 0x7f49400f9d60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f4930009cc0 tx=0x7f49300092d0).ready entity=mon.0 client_cookie=2eed05a65a688910 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:51.973 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.965+0000 7f493effd700 1 -- 172.21.15.5:0/1740013024 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f493000bbf0 con 0x7f49400f9940 2021-03-09T08:55:51.974 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.965+0000 7f493effd700 1 -- 172.21.15.5:0/1740013024 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7f493000bd50 con 0x7f49400f9940 2021-03-09T08:55:51.974 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.965+0000 7f493effd700 1 -- 172.21.15.5:0/1740013024 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4930010560 con 0x7f49400f9940 2021-03-09T08:55:51.974 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f49450fe700 1 -- 172.21.15.5:0/1740013024 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49400f9940 msgr2=0x7f49400f9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:55:51.975 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f49450fe700 1 --2- 172.21.15.5:0/1740013024 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49400f9940 0x7f49400f9d60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f4930009cc0 tx=0x7f49300092d0).stop 2021-03-09T08:55:51.975 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f49450fe700 1 -- 172.21.15.5:0/1740013024 shutdown_connections 2021-03-09T08:55:51.976 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f49450fe700 1 --2- 172.21.15.5:0/1740013024 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49400f9940 0x7f49400f9d60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:51.976 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f49450fe700 1 -- 172.21.15.5:0/1740013024 >> 172.21.15.5:0/1740013024 conn(0x7f49400f5120 msgr2=0x7f49400f7560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:55:51.977 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f49450fe700 1 -- 172.21.15.5:0/1740013024 shutdown_connections 2021-03-09T08:55:51.977 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f49450fe700 1 -- 172.21.15.5:0/1740013024 wait complete. 2021-03-09T08:55:51.981 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f49450fe700 1 Processor -- start 2021-03-09T08:55:51.982 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f49450fe700 1 -- start start 2021-03-09T08:55:51.982 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f49450fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49400f9940 0x7f49401080a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:51.983 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f49450fe700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4940108590 con 0x7f49400f9940 2021-03-09T08:55:51.983 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f493ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49400f9940 0x7f49401080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:51.983 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f493ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49400f9940 0x7f49401080a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46240/0 (socket says 172.21.15.5:46240) 2021-03-09T08:55:51.984 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f493ffff700 1 -- 172.21.15.5:0/970321751 learned_addr learned my addr 172.21.15.5:0/970321751 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:55:51.984 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f493ffff700 1 -- 172.21.15.5:0/970321751 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49401086d0 con 0x7f49400f9940 2021-03-09T08:55:51.985 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f493ffff700 1 --2- 172.21.15.5:0/970321751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49400f9940 0x7f49401080a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4930010a50 tx=0x7f49300092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:51.985 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f493cff9700 1 -- 172.21.15.5:0/970321751 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f493000bbf0 con 0x7f49400f9940 2021-03-09T08:55:51.985 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f49450fe700 1 -- 172.21.15.5:0/970321751 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4940108920 con 0x7f49400f9940 2021-03-09T08:55:51.986 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f493cff9700 1 -- 172.21.15.5:0/970321751 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7f493000bec0 con 0x7f49400f9940 2021-03-09T08:55:51.986 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f493cff9700 1 -- 172.21.15.5:0/970321751 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f49300235d0 con 0x7f49400f9940 2021-03-09T08:55:51.986 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.969+0000 7f49450fe700 1 -- 172.21.15.5:0/970321751 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4940109480 con 0x7f49400f9940 2021-03-09T08:55:51.986 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.973+0000 7f493cff9700 1 -- 172.21.15.5:0/970321751 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f493000f070 con 0x7f49400f9940 2021-03-09T08:55:51.987 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.973+0000 7f493cff9700 1 -- 172.21.15.5:0/970321751 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f493002cb20 con 0x7f49400f9940 2021-03-09T08:55:51.987 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.973+0000 7f49450fe700 1 -- 172.21.15.5:0/970321751 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49400589f0 con 0x7f49400f9940 2021-03-09T08:55:51.987 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:51.981+0000 7f493cff9700 1 -- 172.21.15.5:0/970321751 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f493001e030 con 0x7f49400f9940 2021-03-09T08:55:52.111 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:52.109+0000 7f49450fe700 1 -- 172.21.15.5:0/970321751 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f49400fb7e0 con 0x7f49400f9940 2021-03-09T08:55:52.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.821+0000 7f133b68e700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-09T08:55:52.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.821+0000 7f133b68e700 0 ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev), process ceph-mon, pid 6 2021-03-09T08:55:52.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.821+0000 7f133b68e700 0 pidfile_write: ignore empty --pid-file 2021-03-09T08:55:52.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 0 load: jerasure load: lrc load: isa 2021-03-09T08:55:52.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: RocksDB version: 6.8.1 2021-03-09T08:55:52.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-09T08:55:52.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Compile date Mar 9 2021 2021-03-09T08:55:52.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: DB SUMMARY 2021-03-09T08:55:52.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: CURRENT file: CURRENT 2021-03-09T08:55:52.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-09T08:55:52.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2021-03-09T08:55:52.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2021-03-09T08:55:52.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 85599 ; 2021-03-09T08:55:52.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.error_if_exists: 0 2021-03-09T08:55:52.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.create_if_missing: 0 2021-03-09T08:55:52.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.paranoid_checks: 1 2021-03-09T08:55:52.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.env: 0x5565897511c0 2021-03-09T08:55:52.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.fs: Posix File System 2021-03-09T08:55:52.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.info_log: 0x5565936cf120 2021-03-09T08:55:52.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-09T08:55:52.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.statistics: (nil) 2021-03-09T08:55:52.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.use_fsync: 0 2021-03-09T08:55:52.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_log_file_size: 0 2021-03-09T08:55:52.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-09T08:55:52.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-09T08:55:52.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-09T08:55:52.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-09T08:55:52.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.allow_fallocate: 1 2021-03-09T08:55:52.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-09T08:55:52.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-09T08:55:52.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.use_direct_reads: 0 2021-03-09T08:55:52.121 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-09T08:55:52.121 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-09T08:55:52.121 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.db_log_dir: 2021-03-09T08:55:52.121 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2021-03-09T08:55:52.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-09T08:55:52.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_subcompactions: 1 2021-03-09T08:55:52.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_background_flushes: -1 2021-03-09T08:55:52.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-09T08:55:52.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-09T08:55:52.123 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-09T08:55:52.123 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-09T08:55:52.123 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-09T08:55:52.123 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-09T08:55:52.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-09T08:55:52.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.write_buffer_manager: 0x556593723cb0 2021-03-09T08:55:52.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-09T08:55:52.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-09T08:55:52.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-09T08:55:52.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-09T08:55:52.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-09T08:55:52.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-09T08:55:52.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-09T08:55:52.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-09T08:55:52.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-09T08:55:52.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.unordered_write: 0 2021-03-09T08:55:52.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-09T08:55:52.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-09T08:55:52.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-09T08:55:52.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-09T08:55:52.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.row_cache: None 2021-03-09T08:55:52.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.wal_filter: None 2021-03-09T08:55:52.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-09T08:55:52.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-09T08:55:52.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.preserve_deletes: 0 2021-03-09T08:55:52.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.two_write_queues: 0 2021-03-09T08:55:52.130 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-09T08:55:52.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.atomic_flush: 0 2021-03-09T08:55:52.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-09T08:55:52.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-09T08:55:52.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-09T08:55:52.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.log_readahead_size: 0 2021-03-09T08:55:52.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-09T08:55:52.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_background_jobs: 2 2021-03-09T08:55:52.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_background_compactions: -1 2021-03-09T08:55:52.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-09T08:55:52.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-09T08:55:52.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-09T08:55:52.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-09T08:55:52.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-09T08:55:52.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-09T08:55:52.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-09T08:55:52.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-09T08:55:52.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_open_files: -1 2021-03-09T08:55:52.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-09T08:55:52.135 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-09T08:55:52.135 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-09T08:55:52.135 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-09T08:55:52.135 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Compression algorithms supported: 2021-03-09T08:55:52.135 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-09T08:55:52.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: kZSTD supported: 0 2021-03-09T08:55:52.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: kXpressCompression supported: 0 2021-03-09T08:55:52.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-09T08:55:52.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: kLZ4Compression supported: 1 2021-03-09T08:55:52.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: kBZip2Compression supported: 0 2021-03-09T08:55:52.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: kZlibCompression supported: 1 2021-03-09T08:55:52.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: kSnappyCompression supported: 1 2021-03-09T08:55:52.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-09T08:55:52.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2021-03-09T08:55:52.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-09T08:55:52.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-09T08:55:52.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.merge_operator: 2021-03-09T08:55:52.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compaction_filter: None 2021-03-09T08:55:52.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-09T08:55:52.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-09T08:55:52.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-09T08:55:52.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55658b472230) 2021-03-09T08:55:52.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: cache_index_and_filter_blocks: 1 2021-03-09T08:55:52.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-09T08:55:52.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-09T08:55:52.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: pin_top_level_index_and_filter: 1 2021-03-09T08:55:52.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: index_type: 0 2021-03-09T08:55:52.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: data_block_index_type: 0 2021-03-09T08:55:52.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: index_shortening: 1 2021-03-09T08:55:52.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: data_block_hash_table_util_ratio: 0.750000 2021-03-09T08:55:52.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: hash_index_allow_collision: 1 2021-03-09T08:55:52.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: checksum: 1 2021-03-09T08:55:52.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: no_block_cache: 0 2021-03-09T08:55:52.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: block_cache: 0x5565936aa510 2021-03-09T08:55:52.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: block_cache_name: BinnedLRUCache 2021-03-09T08:55:52.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: block_cache_options: 2021-03-09T08:55:52.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: capacity : 536870912 2021-03-09T08:55:52.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: num_shard_bits : 4 2021-03-09T08:55:52.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: strict_capacity_limit : 0 2021-03-09T08:55:52.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: high_pri_pool_ratio: 0.000 2021-03-09T08:55:52.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: block_cache_compressed: (nil) 2021-03-09T08:55:52.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: persistent_cache: (nil) 2021-03-09T08:55:52.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: block_size: 4096 2021-03-09T08:55:52.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: block_size_deviation: 10 2021-03-09T08:55:52.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: block_restart_interval: 16 2021-03-09T08:55:52.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: index_block_restart_interval: 1 2021-03-09T08:55:52.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: metadata_block_size: 4096 2021-03-09T08:55:52.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: partition_filters: 0 2021-03-09T08:55:52.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: use_delta_encoding: 1 2021-03-09T08:55:52.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-09T08:55:52.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: whole_key_filtering: 1 2021-03-09T08:55:52.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: verify_compression: 0 2021-03-09T08:55:52.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: read_amp_bytes_per_bit: 0 2021-03-09T08:55:52.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: format_version: 2 2021-03-09T08:55:52.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: enable_index_compression: 1 2021-03-09T08:55:52.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: block_align: 0 2021-03-09T08:55:52.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-09T08:55:52.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-09T08:55:52.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compression: NoCompression 2021-03-09T08:55:52.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-09T08:55:52.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-09T08:55:52.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-09T08:55:52.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.num_levels: 7 2021-03-09T08:55:52.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-09T08:55:52.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-09T08:55:52.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-09T08:55:52.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-09T08:55:52.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-09T08:55:52.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-09T08:55:52.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-09T08:55:52.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-09T08:55:52.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-09T08:55:52.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-09T08:55:52.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-09T08:55:52.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-09T08:55:52.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-09T08:55:52.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-09T08:55:52.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-09T08:55:52.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-09T08:55:52.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-09T08:55:52.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-09T08:55:52.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-09T08:55:52.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-09T08:55:52.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-09T08:55:52.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-09T08:55:52.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-09T08:55:52.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-09T08:55:52.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-09T08:55:52.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-09T08:55:52.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-09T08:55:52.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-09T08:55:52.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-09T08:55:52.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-09T08:55:52.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-09T08:55:52.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-09T08:55:52.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-09T08:55:52.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-09T08:55:52.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-09T08:55:52.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-09T08:55:52.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-09T08:55:52.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-09T08:55:52.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-09T08:55:52.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-09T08:55:52.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-09T08:55:52.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-09T08:55:52.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-09T08:55:52.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-09T08:55:52.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-09T08:55:52.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-09T08:55:52.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-09T08:55:52.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.table_properties_collectors: 2021-03-09T08:55:52.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.inplace_update_support: 0 2021-03-09T08:55:52.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-09T08:55:52.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-09T08:55:52.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-09T08:55:52.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-09T08:55:52.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.bloom_locality: 0 2021-03-09T08:55:52.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.max_successive_merges: 0 2021-03-09T08:55:52.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-09T08:55:52.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-09T08:55:52.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-09T08:55:52.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-09T08:55:52.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.ttl: 2592000 2021-03-09T08:55:52.161 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:52.113+0000 7f493cff9700 1 -- 172.21.15.5:0/970321751 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f493001ab50 con 0x7f49400f9940 2021-03-09T08:55:52.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.825+0000 7f133b68e700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-09T08:55:52.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.829+0000 7f133b68e700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-09T08:55:52.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.829+0000 7f133b68e700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2021-03-09T08:55:52.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.829+0000 7f133b68e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280151833341, "job": 1, "event": "recovery_started", "log_files": [6]} 2021-03-09T08:55:52.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.829+0000 7f133b68e700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2021-03-09T08:55:52.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.829+0000 7f133b68e700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-09T08:55:52.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.829+0000 7f133b68e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280151834522, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 82330, "table_properties": {"data_size": 80599, "index_size": 336, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 9383, "raw_average_key_size": 46, "raw_value_size": 75219, "raw_average_value_size": 372, "num_data_blocks": 11, "num_entries": 202, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1615280151, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-09T08:55:52.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.829+0000 7f133b68e700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2021-03-09T08:55:52.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.829+0000 7f133b68e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280151835337, "job": 1, "event": "recovery_finished"} 2021-03-09T08:55:52.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.833+0000 7f133b68e700 4 rocksdb: DB pointer 0x55659376f800 2021-03-09T08:55:52.165 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.833+0000 7f132a69e700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-09T08:55:52.165 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.833+0000 7f132a69e700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-09T08:55:52.165 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: ** DB Stats ** 2021-03-09T08:55:52.165 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T08:55:52.165 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-09T08:55:52.166 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-09T08:55:52.166 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T08:55:52.166 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-09T08:55:52.166 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-09T08:55:52.166 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T08:55:52.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: ** Compaction Stats [default] ** 2021-03-09T08:55:52.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T08:55:52.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T08:55:52.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: L0 2/0 82.05 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 95.9 0.00 0.00 1 0.001 0 0 2021-03-09T08:55:52.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Sum 2/0 82.05 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 95.9 0.00 0.00 1 0.001 0 0 2021-03-09T08:55:52.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 95.9 0.00 0.00 1 0.001 0 0 2021-03-09T08:55:52.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: ** Compaction Stats [default] ** 2021-03-09T08:55:52.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T08:55:52.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T08:55:52.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 95.9 0.00 0.00 1 0.001 0 0 2021-03-09T08:55:52.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T08:55:52.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T08:55:52.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T08:55:52.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T08:55:52.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T08:55:52.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T08:55:52.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Cumulative compaction: 0.00 GB write, 12.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T08:55:52.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Interval compaction: 0.00 GB write, 12.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T08:55:52.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T08:55:52.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T08:55:52.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: ** Compaction Stats [default] ** 2021-03-09T08:55:52.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T08:55:52.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T08:55:52.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: L0 2/0 82.05 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 95.9 0.00 0.00 1 0.001 0 0 2021-03-09T08:55:52.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Sum 2/0 82.05 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 95.9 0.00 0.00 1 0.001 0 0 2021-03-09T08:55:52.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T08:55:52.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: ** Compaction Stats [default] ** 2021-03-09T08:55:52.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T08:55:52.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T08:55:52.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 95.9 0.00 0.00 1 0.001 0 0 2021-03-09T08:55:52.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T08:55:52.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T08:55:52.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T08:55:52.174 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T08:55:52.174 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T08:55:52.174 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T08:55:52.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Cumulative compaction: 0.00 GB write, 12.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T08:55:52.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T08:55:52.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T08:55:52.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T08:55:52.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: debug 2021-03-09T08:55:51.833+0000 7f133b68e700 0 starting mon.a rank 0 at public addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] at bind addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:55:52.176 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:51 smithi005 bash[12738]: audit 2021-03-09T08:55:48.689061+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.5:0/2189108485' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:55:52.177 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:52.117+0000 7f49450fe700 1 -- 172.21.15.5:0/970321751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49400f9940 msgr2=0x7f49401080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:52.117+0000 7f49450fe700 1 --2- 172.21.15.5:0/970321751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49400f9940 0x7f49401080a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4930010a50 tx=0x7f49300092d0).stop 2021-03-09T08:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:52.117+0000 7f49450fe700 1 -- 172.21.15.5:0/970321751 shutdown_connections 2021-03-09T08:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:52.117+0000 7f49450fe700 1 --2- 172.21.15.5:0/970321751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f49400f9940 0x7f49401080a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:52.117+0000 7f49450fe700 1 -- 172.21.15.5:0/970321751 >> 172.21.15.5:0/970321751 conn(0x7f49400f5120 msgr2=0x7f49400f67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:55:52.179 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:52.117+0000 7f49450fe700 1 -- 172.21.15.5:0/970321751 shutdown_connections 2021-03-09T08:55:52.179 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:52.117+0000 7f49450fe700 1 -- 172.21.15.5:0/970321751 wait complete. 2021-03-09T08:55:52.867 INFO:teuthology.orchestra.run.smithi005.stderr:Wrote config to /etc/ceph/ceph.conf 2021-03-09T08:55:52.868 INFO:teuthology.orchestra.run.smithi005.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2021-03-09T08:55:52.868 INFO:teuthology.orchestra.run.smithi005.stderr:Creating mgr... 2021-03-09T08:55:52.869 INFO:teuthology.orchestra.run.smithi005.stderr:Verifying port 9283 ... 2021-03-09T08:55:52.871 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl daemon-reload 2021-03-09T08:55:53.052 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl stop ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mgr.y 2021-03-09T08:55:53.065 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl reset-failed ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mgr.y 2021-03-09T08:55:53.071 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Failed to reset failed state of unit ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mgr.y.service: Unit ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mgr.y.service not loaded. 2021-03-09T08:55:53.072 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl enable ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mgr.y 2021-03-09T08:55:53.080 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Created symlink /etc/systemd/system/ceph-252abbe0-80b5-11eb-907b-001a4aab830c.target.wants/ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mgr.y.service -> /etc/systemd/system/ceph-252abbe0-80b5-11eb-907b-001a4aab830c@.service. 2021-03-09T08:55:53.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:52 smithi005 bash[12738]: cluster 2021-03-09T08:55:51.843245+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2021-03-09T08:55:53.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:52 smithi005 bash[12738]: cluster 2021-03-09T08:55:51.843352+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0]} 2021-03-09T08:55:53.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:52 smithi005 bash[12738]: cluster 2021-03-09T08:55:51.843437+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2021-03-09T08:55:53.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:52 smithi005 bash[12738]: cluster 2021-03-09T08:55:51.843496+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-03-09T08:55:53.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:52 smithi005 bash[12738]: cluster 2021-03-09T08:55:51.844020+0000 mon.a (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2021-03-09T08:55:53.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:52 smithi005 bash[12738]: audit 2021-03-09T08:55:52.119069+0000 mon.a (mon.0) 6 : audit [INF] from='client.? 172.21.15.5:0/970321751' entity='client.admin' 2021-03-09T08:55:53.259 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: systemctl start ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mgr.y 2021-03-09T08:55:53.270 INFO:teuthology.orchestra.run.smithi005.stderr:firewalld does not appear to be present 2021-03-09T08:55:53.271 INFO:teuthology.orchestra.run.smithi005.stderr:Not possible to enable service . firewalld.service is not available 2021-03-09T08:55:53.271 INFO:teuthology.orchestra.run.smithi005.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-03-09T08:55:53.272 INFO:teuthology.orchestra.run.smithi005.stderr:firewalld does not appear to be present 2021-03-09T08:55:53.272 INFO:teuthology.orchestra.run.smithi005.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2021-03-09T08:55:53.272 INFO:teuthology.orchestra.run.smithi005.stderr:Waiting for mgr to start... 2021-03-09T08:55:53.272 INFO:teuthology.orchestra.run.smithi005.stderr:Waiting for mgr... 2021-03-09T08:55:53.273 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 status -f json-pretty 2021-03-09T08:55:54.753 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.749+0000 7fd352846700 1 Processor -- start 2021-03-09T08:55:54.753 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.749+0000 7fd352846700 1 -- start start 2021-03-09T08:55:54.753 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd352846700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd34c06dc10 0x7fd34c0fe9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:54.754 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd352846700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c0fa3c0 con 0x7fd34c06dc10 2021-03-09T08:55:54.754 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd351844700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd34c06dc10 0x7fd34c0fe9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:54.755 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd351844700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd34c06dc10 0x7fd34c0fe9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46246/0 (socket says 172.21.15.5:46246) 2021-03-09T08:55:54.755 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd351844700 1 -- 172.21.15.5:0/3441283505 learned_addr learned my addr 172.21.15.5:0/3441283505 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:55:54.755 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd351844700 1 -- 172.21.15.5:0/3441283505 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c0feec0 con 0x7fd34c06dc10 2021-03-09T08:55:54.757 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd351844700 1 --2- 172.21.15.5:0/3441283505 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd34c06dc10 0x7fd34c0fe9d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd34800b950 tx=0x7fd348006ee0).ready entity=mon.0 client_cookie=8230754f6e0ba6eb server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:54.757 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd350842700 1 -- 172.21.15.5:0/3441283505 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd3480098d0 con 0x7fd34c06dc10 2021-03-09T08:55:54.758 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd350842700 1 -- 172.21.15.5:0/3441283505 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7fd348009e90 con 0x7fd34c06dc10 2021-03-09T08:55:54.758 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd350842700 1 -- 172.21.15.5:0/3441283505 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd348011d10 con 0x7fd34c06dc10 2021-03-09T08:55:54.758 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd352846700 1 -- 172.21.15.5:0/3441283505 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd34c06dc10 msgr2=0x7fd34c0fe9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:55:54.758 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd352846700 1 --2- 172.21.15.5:0/3441283505 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd34c06dc10 0x7fd34c0fe9d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd34800b950 tx=0x7fd348006ee0).stop 2021-03-09T08:55:54.758 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd352846700 1 -- 172.21.15.5:0/3441283505 shutdown_connections 2021-03-09T08:55:54.759 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd352846700 1 --2- 172.21.15.5:0/3441283505 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd34c06dc10 0x7fd34c0fe9d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:54.759 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd352846700 1 -- 172.21.15.5:0/3441283505 >> 172.21.15.5:0/3441283505 conn(0x7fd34c0f4030 msgr2=0x7fd34c0f6450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:55:54.759 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd352846700 1 -- 172.21.15.5:0/3441283505 shutdown_connections 2021-03-09T08:55:54.759 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd352846700 1 -- 172.21.15.5:0/3441283505 wait complete. 2021-03-09T08:55:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd352846700 1 Processor -- start 2021-03-09T08:55:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd352846700 1 -- start start 2021-03-09T08:55:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd352846700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd34c06dc10 0x7fd34c10a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd352846700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c10a710 con 0x7fd34c06dc10 2021-03-09T08:55:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd351844700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd34c06dc10 0x7fd34c10a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:54.761 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd351844700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd34c06dc10 0x7fd34c10a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46248/0 (socket says 172.21.15.5:46248) 2021-03-09T08:55:54.761 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.753+0000 7fd351844700 1 -- 172.21.15.5:0/485113135 learned_addr learned my addr 172.21.15.5:0/485113135 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:55:54.761 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.757+0000 7fd351844700 1 -- 172.21.15.5:0/485113135 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c10a850 con 0x7fd34c06dc10 2021-03-09T08:55:54.761 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.757+0000 7fd351844700 1 --2- 172.21.15.5:0/485113135 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd34c06dc10 0x7fd34c10a220 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd348000c00 tx=0x7fd348006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:54.761 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.757+0000 7fd33a7fc700 1 -- 172.21.15.5:0/485113135 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd348009d30 con 0x7fd34c06dc10 2021-03-09T08:55:54.762 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.757+0000 7fd352846700 1 -- 172.21.15.5:0/485113135 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34c10aaa0 con 0x7fd34c06dc10 2021-03-09T08:55:54.763 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.757+0000 7fd33a7fc700 1 -- 172.21.15.5:0/485113135 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7fd348011770 con 0x7fd34c06dc10 2021-03-09T08:55:54.763 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.757+0000 7fd33a7fc700 1 -- 172.21.15.5:0/485113135 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd34801bb80 con 0x7fd34c06dc10 2021-03-09T08:55:54.763 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.757+0000 7fd33a7fc700 1 -- 172.21.15.5:0/485113135 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7fd34800f070 con 0x7fd34c06dc10 2021-03-09T08:55:54.764 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.757+0000 7fd352846700 1 -- 172.21.15.5:0/485113135 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34c10b600 con 0x7fd34c06dc10 2021-03-09T08:55:54.764 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.757+0000 7fd33a7fc700 1 -- 172.21.15.5:0/485113135 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd348010d60 con 0x7fd34c06dc10 2021-03-09T08:55:54.764 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.757+0000 7fd352846700 1 -- 172.21.15.5:0/485113135 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34c0589f0 con 0x7fd34c06dc10 2021-03-09T08:55:54.764 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.761+0000 7fd33a7fc700 1 -- 172.21.15.5:0/485113135 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7fd348016070 con 0x7fd34c06dc10 2021-03-09T08:55:54.895 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.893+0000 7fd352846700 1 -- 172.21.15.5:0/485113135 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd34c0ffad0 con 0x7fd34c06dc10 2021-03-09T08:55:54.904 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.897+0000 7fd33a7fc700 1 -- 172.21.15.5:0/485113135 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1221 (secure 0 0 0) 0x7fd348025970 con 0x7fd34c06dc10 2021-03-09T08:55:54.904 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.904 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: { 2021-03-09T08:55:54.904 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "fsid": "252abbe0-80b5-11eb-907b-001a4aab830c", 2021-03-09T08:55:54.905 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "health": { 2021-03-09T08:55:54.905 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-03-09T08:55:54.905 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "checks": {}, 2021-03-09T08:55:54.905 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "mutes": [] 2021-03-09T08:55:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:55:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-03-09T08:55:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum": [ 2021-03-09T08:55:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 0 2021-03-09T08:55:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2021-03-09T08:55:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum_names": [ 2021-03-09T08:55:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "a" 2021-03-09T08:55:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2021-03-09T08:55:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum_age": 3, 2021-03-09T08:55:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "monmap": { 2021-03-09T08:55:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-09T08:55:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2021-03-09T08:55:54.909 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_mons": 1 2021-03-09T08:55:54.909 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:55:54.909 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osdmap": { 2021-03-09T08:55:54.909 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-09T08:55:54.909 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_osds": 0, 2021-03-09T08:55:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-03-09T08:55:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-03-09T08:55:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-03-09T08:55:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-03-09T08:55:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-03-09T08:55:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:55:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "pgmap": { 2021-03-09T08:55:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-03-09T08:55:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-03-09T08:55:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_pools": 0, 2021-03-09T08:55:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_objects": 0, 2021-03-09T08:55:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-03-09T08:55:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-03-09T08:55:54.913 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-03-09T08:55:54.913 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_total": 0 2021-03-09T08:55:54.913 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:55:54.913 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "fsmap": { 2021-03-09T08:55:54.913 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-09T08:55:54.914 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "by_rank": [], 2021-03-09T08:55:54.914 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "up:standby": 0 2021-03-09T08:55:54.914 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:55:54.915 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "mgrmap": { 2021-03-09T08:55:54.915 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "available": false, 2021-03-09T08:55:54.915 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-03-09T08:55:54.916 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "modules": [ 2021-03-09T08:55:54.916 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "iostat", 2021-03-09T08:55:54.916 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "restful" 2021-03-09T08:55:54.916 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2021-03-09T08:55:54.916 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "services": {} 2021-03-09T08:55:54.917 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:55:54.917 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "servicemap": { 2021-03-09T08:55:54.917 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-09T08:55:54.917 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "modified": "2021-03-09T08:55:44.624898+0000", 2021-03-09T08:55:54.918 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "services": {} 2021-03-09T08:55:54.918 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:55:54.918 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "progress_events": {} 2021-03-09T08:55:54.918 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: } 2021-03-09T08:55:54.919 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.897+0000 7fd352846700 1 -- 172.21.15.5:0/485113135 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd34c06dc10 msgr2=0x7fd34c10a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:55:54.919 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.897+0000 7fd352846700 1 --2- 172.21.15.5:0/485113135 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd34c06dc10 0x7fd34c10a220 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd348000c00 tx=0x7fd348006ee0).stop 2021-03-09T08:55:54.919 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.897+0000 7fd352846700 1 -- 172.21.15.5:0/485113135 shutdown_connections 2021-03-09T08:55:54.919 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.897+0000 7fd352846700 1 --2- 172.21.15.5:0/485113135 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd34c06dc10 0x7fd34c10a220 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:54.919 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.897+0000 7fd352846700 1 -- 172.21.15.5:0/485113135 >> 172.21.15.5:0/485113135 conn(0x7fd34c0f4030 msgr2=0x7fd34c0f4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:55:54.920 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.897+0000 7fd352846700 1 -- 172.21.15.5:0/485113135 shutdown_connections 2021-03-09T08:55:54.920 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:54.897+0000 7fd352846700 1 -- 172.21.15.5:0/485113135 wait complete. 2021-03-09T08:55:55.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:54 smithi005 bash[12738]: audit 2021-03-09T08:55:54.899540+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.5:0/485113135' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-03-09T08:55:55.602 INFO:teuthology.orchestra.run.smithi005.stderr:mgr not available, waiting (1/15)... 2021-03-09T08:55:57.604 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 status -f json-pretty 2021-03-09T08:55:58.750 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.749+0000 7f26dd43d700 1 Processor -- start 2021-03-09T08:55:58.750 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.749+0000 7f26dd43d700 1 -- start start 2021-03-09T08:55:58.751 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.749+0000 7f26dd43d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26d806dc10 0x7f26d80fc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:58.751 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.749+0000 7f26dd43d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d80fcca0 con 0x7f26d806dc10 2021-03-09T08:55:58.751 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.749+0000 7f26d6ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26d806dc10 0x7f26d80fc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.749+0000 7f26d6ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26d806dc10 0x7f26d80fc7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46250/0 (socket says 172.21.15.5:46250) 2021-03-09T08:55:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.749+0000 7f26d6ffd700 1 -- 172.21.15.5:0/2430966930 learned_addr learned my addr 172.21.15.5:0/2430966930 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:55:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.749+0000 7f26d6ffd700 1 -- 172.21.15.5:0/2430966930 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d80fcde0 con 0x7f26d806dc10 2021-03-09T08:55:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.749+0000 7f26d6ffd700 1 --2- 172.21.15.5:0/2430966930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26d806dc10 0x7f26d80fc7b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f26c800b950 tx=0x7f26c8006ee0).ready entity=mon.0 client_cookie=6ea830e30da4045e server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.749+0000 7f26d67fc700 1 -- 172.21.15.5:0/2430966930 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26c80098d0 con 0x7f26d806dc10 2021-03-09T08:55:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.749+0000 7f26d67fc700 1 -- 172.21.15.5:0/2430966930 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7f26c8009e90 con 0x7f26d806dc10 2021-03-09T08:55:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.749+0000 7f26d67fc700 1 -- 172.21.15.5:0/2430966930 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26c8011d10 con 0x7f26d806dc10 2021-03-09T08:55:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26dd43d700 1 -- 172.21.15.5:0/2430966930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26d806dc10 msgr2=0x7f26d80fc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:55:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26dd43d700 1 --2- 172.21.15.5:0/2430966930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26d806dc10 0x7f26d80fc7b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f26c800b950 tx=0x7f26c8006ee0).stop 2021-03-09T08:55:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26dd43d700 1 -- 172.21.15.5:0/2430966930 shutdown_connections 2021-03-09T08:55:58.756 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26dd43d700 1 --2- 172.21.15.5:0/2430966930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26d806dc10 0x7f26d80fc7b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:58.756 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26dd43d700 1 -- 172.21.15.5:0/2430966930 >> 172.21.15.5:0/2430966930 conn(0x7f26d80f41f0 msgr2=0x7f26d80f6630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:55:58.756 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26dd43d700 1 -- 172.21.15.5:0/2430966930 shutdown_connections 2021-03-09T08:55:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26dd43d700 1 -- 172.21.15.5:0/2430966930 wait complete. 2021-03-09T08:55:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26dd43d700 1 Processor -- start 2021-03-09T08:55:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26dd43d700 1 -- start start 2021-03-09T08:55:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26dd43d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26d806dc10 0x7f26d81081f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26dd43d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d81086e0 con 0x7f26d806dc10 2021-03-09T08:55:58.758 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26d6ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26d806dc10 0x7f26d81081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:58.758 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26d6ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26d806dc10 0x7f26d81081f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46252/0 (socket says 172.21.15.5:46252) 2021-03-09T08:55:58.759 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26d6ffd700 1 -- 172.21.15.5:0/1738702149 learned_addr learned my addr 172.21.15.5:0/1738702149 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:55:58.760 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26d6ffd700 1 -- 172.21.15.5:0/1738702149 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d8108820 con 0x7f26d806dc10 2021-03-09T08:55:58.760 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26d6ffd700 1 --2- 172.21.15.5:0/1738702149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26d806dc10 0x7f26d81081f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f26c8000c00 tx=0x7f26c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:58.760 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26cf7fe700 1 -- 172.21.15.5:0/1738702149 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26c8009d30 con 0x7f26d806dc10 2021-03-09T08:55:58.760 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26dd43d700 1 -- 172.21.15.5:0/1738702149 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d8108a70 con 0x7f26d806dc10 2021-03-09T08:55:58.761 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26dd43d700 1 -- 172.21.15.5:0/1738702149 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d81095d0 con 0x7f26d806dc10 2021-03-09T08:55:58.761 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26cf7fe700 1 -- 172.21.15.5:0/1738702149 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7f26c8011770 con 0x7f26d806dc10 2021-03-09T08:55:58.761 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26cf7fe700 1 -- 172.21.15.5:0/1738702149 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26c801ba30 con 0x7f26d806dc10 2021-03-09T08:55:58.761 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26cf7fe700 1 -- 172.21.15.5:0/1738702149 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f26c800f070 con 0x7f26d806dc10 2021-03-09T08:55:58.761 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.753+0000 7f26cf7fe700 1 -- 172.21.15.5:0/1738702149 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f26c8024780 con 0x7f26d806dc10 2021-03-09T08:55:58.762 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.757+0000 7f26dd43d700 1 -- 172.21.15.5:0/1738702149 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26bc004da0 con 0x7f26d806dc10 2021-03-09T08:55:58.762 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.761+0000 7f26cf7fe700 1 -- 172.21.15.5:0/1738702149 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f26c8016070 con 0x7f26d806dc10 2021-03-09T08:55:58.894 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.893+0000 7f26dd43d700 1 -- 172.21.15.5:0/1738702149 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f26bc005050 con 0x7f26d806dc10 2021-03-09T08:55:58.901 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.893+0000 7f26cf7fe700 1 -- 172.21.15.5:0/1738702149 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1221 (secure 0 0 0) 0x7f26c801b410 con 0x7f26d806dc10 2021-03-09T08:55:58.901 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.902 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: { 2021-03-09T08:55:58.902 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "fsid": "252abbe0-80b5-11eb-907b-001a4aab830c", 2021-03-09T08:55:58.902 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "health": { 2021-03-09T08:55:58.902 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-03-09T08:55:58.903 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "checks": {}, 2021-03-09T08:55:58.903 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "mutes": [] 2021-03-09T08:55:58.903 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:55:58.903 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-03-09T08:55:58.903 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum": [ 2021-03-09T08:55:58.904 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 0 2021-03-09T08:55:58.904 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2021-03-09T08:55:58.904 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum_names": [ 2021-03-09T08:55:58.904 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "a" 2021-03-09T08:55:58.904 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2021-03-09T08:55:58.905 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum_age": 7, 2021-03-09T08:55:58.905 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "monmap": { 2021-03-09T08:55:58.905 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-09T08:55:58.905 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2021-03-09T08:55:58.906 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_mons": 1 2021-03-09T08:55:58.906 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:55:58.906 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osdmap": { 2021-03-09T08:55:58.906 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-09T08:55:58.906 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_osds": 0, 2021-03-09T08:55:58.907 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-03-09T08:55:58.907 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-03-09T08:55:58.907 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-03-09T08:55:58.907 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-03-09T08:55:58.907 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-03-09T08:55:58.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:55:58.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "pgmap": { 2021-03-09T08:55:58.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-03-09T08:55:58.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-03-09T08:55:58.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_pools": 0, 2021-03-09T08:55:58.909 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_objects": 0, 2021-03-09T08:55:58.909 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-03-09T08:55:58.909 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-03-09T08:55:58.909 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-03-09T08:55:58.909 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_total": 0 2021-03-09T08:55:58.910 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:55:58.910 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "fsmap": { 2021-03-09T08:55:58.910 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-09T08:55:58.910 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "by_rank": [], 2021-03-09T08:55:58.910 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "up:standby": 0 2021-03-09T08:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "mgrmap": { 2021-03-09T08:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "available": false, 2021-03-09T08:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-03-09T08:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "modules": [ 2021-03-09T08:55:58.912 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "iostat", 2021-03-09T08:55:58.912 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "restful" 2021-03-09T08:55:58.912 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2021-03-09T08:55:58.912 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "services": {} 2021-03-09T08:55:58.913 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:55:58.913 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "servicemap": { 2021-03-09T08:55:58.913 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-09T08:55:58.913 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "modified": "2021-03-09T08:55:44.624898+0000", 2021-03-09T08:55:58.913 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "services": {} 2021-03-09T08:55:58.914 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:55:58.914 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "progress_events": {} 2021-03-09T08:55:58.914 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: } 2021-03-09T08:55:58.914 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.897+0000 7f26dd43d700 1 -- 172.21.15.5:0/1738702149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26d806dc10 msgr2=0x7f26d81081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:55:58.914 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.897+0000 7f26dd43d700 1 --2- 172.21.15.5:0/1738702149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26d806dc10 0x7f26d81081f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f26c8000c00 tx=0x7f26c8006ee0).stop 2021-03-09T08:55:58.915 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.897+0000 7f26dd43d700 1 -- 172.21.15.5:0/1738702149 shutdown_connections 2021-03-09T08:55:58.915 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.897+0000 7f26dd43d700 1 --2- 172.21.15.5:0/1738702149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26d806dc10 0x7f26d81081f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:58.915 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.897+0000 7f26dd43d700 1 -- 172.21.15.5:0/1738702149 >> 172.21.15.5:0/1738702149 conn(0x7f26d80f41f0 msgr2=0x7f26d80f4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:55:58.915 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.897+0000 7f26dd43d700 1 -- 172.21.15.5:0/1738702149 shutdown_connections 2021-03-09T08:55:58.915 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:55:58.897+0000 7f26dd43d700 1 -- 172.21.15.5:0/1738702149 wait complete. 2021-03-09T08:55:59.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:55:58 smithi005 bash[12738]: audit 2021-03-09T08:55:58.898283+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? 172.21.15.5:0/1738702149' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-03-09T08:55:59.549 INFO:teuthology.orchestra.run.smithi005.stderr:mgr not available, waiting (2/15)... 2021-03-09T08:56:01.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:01 smithi005 bash[12738]: cluster 2021-03-09T08:56:00.962229+0000 mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2021-03-09T08:56:01.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:01 smithi005 bash[12738]: cluster 2021-03-09T08:56:00.968854+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00683171s) 2021-03-09T08:56:01.365 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:01 smithi005 bash[12738]: audit 2021-03-09T08:56:00.971570+0000 mon.a (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.5:0/1489707571' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-03-09T08:56:01.365 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:01 smithi005 bash[12738]: audit 2021-03-09T08:56:00.972078+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.5:0/1489707571' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-03-09T08:56:01.365 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:01 smithi005 bash[12738]: audit 2021-03-09T08:56:00.972721+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.5:0/1489707571' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-03-09T08:56:01.365 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:01 smithi005 bash[12738]: audit 2021-03-09T08:56:00.973332+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.5:0/1489707571' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-03-09T08:56:01.365 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:01 smithi005 bash[12738]: audit 2021-03-09T08:56:00.973763+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.5:0/1489707571' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-03-09T08:56:01.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:01 smithi005 bash[12738]: cluster 2021-03-09T08:56:00.978319+0000 mon.a (mon.0) 16 : cluster [INF] Manager daemon y is now available 2021-03-09T08:56:01.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:01 smithi005 bash[12738]: audit 2021-03-09T08:56:00.994097+0000 mon.a (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.5:0/1489707571' entity='mgr.y' 2021-03-09T08:56:01.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:01 smithi005 bash[12738]: audit 2021-03-09T08:56:01.001072+0000 mon.a (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.5:0/1489707571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-09T08:56:01.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:01 smithi005 bash[12738]: audit 2021-03-09T08:56:01.003025+0000 mon.a (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.5:0/1489707571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-09T08:56:01.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:01 smithi005 bash[12738]: audit 2021-03-09T08:56:01.011936+0000 mon.a (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.5:0/1489707571' entity='mgr.y' 2021-03-09T08:56:01.551 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 status -f json-pretty 2021-03-09T08:56:02.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:02 smithi005 bash[12738]: audit 2021-03-09T08:56:01.015049+0000 mon.a (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.5:0/1489707571' entity='mgr.y' 2021-03-09T08:56:02.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:02 smithi005 bash[12738]: cluster 2021-03-09T08:56:01.974410+0000 mon.a (mon.0) 22 : cluster [DBG] mgrmap e3: y(active, since 1.01238s) 2021-03-09T08:56:02.727 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.725+0000 7f1c8023a700 1 Processor -- start 2021-03-09T08:56:02.728 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.725+0000 7f1c8023a700 1 -- start start 2021-03-09T08:56:02.728 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.725+0000 7f1c8023a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c780fbda0 0x7f1c780fc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:02.728 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.725+0000 7f1c8023a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c780fc6b0 con 0x7f1c780fbda0 2021-03-09T08:56:02.728 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.725+0000 7f1c7dfd6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c780fbda0 0x7f1c780fc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:02.729 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.725+0000 7f1c7dfd6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c780fbda0 0x7f1c780fc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46272/0 (socket says 172.21.15.5:46272) 2021-03-09T08:56:02.729 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.725+0000 7f1c7dfd6700 1 -- 172.21.15.5:0/3762842992 learned_addr learned my addr 172.21.15.5:0/3762842992 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:02.729 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.725+0000 7f1c7dfd6700 1 -- 172.21.15.5:0/3762842992 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c780fc7f0 con 0x7f1c780fbda0 2021-03-09T08:56:02.732 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c7dfd6700 1 --2- 172.21.15.5:0/3762842992 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c780fbda0 0x7f1c780fc1c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f1c7400b950 tx=0x7f1c74006ee0).ready entity=mon.0 client_cookie=47fe3eb66996d4b3 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:02.733 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c7cfd4700 1 -- 172.21.15.5:0/3762842992 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c74009840 con 0x7f1c780fbda0 2021-03-09T08:56:02.733 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c7cfd4700 1 -- 172.21.15.5:0/3762842992 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7f1c74009e00 con 0x7f1c780fbda0 2021-03-09T08:56:02.734 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c7cfd4700 1 -- 172.21.15.5:0/3762842992 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c74011c80 con 0x7f1c780fbda0 2021-03-09T08:56:02.734 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c8023a700 1 -- 172.21.15.5:0/3762842992 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c780fbda0 msgr2=0x7f1c780fc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:02.734 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c8023a700 1 --2- 172.21.15.5:0/3762842992 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c780fbda0 0x7f1c780fc1c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f1c7400b950 tx=0x7f1c74006ee0).stop 2021-03-09T08:56:02.734 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c8023a700 1 -- 172.21.15.5:0/3762842992 shutdown_connections 2021-03-09T08:56:02.735 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c8023a700 1 --2- 172.21.15.5:0/3762842992 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c780fbda0 0x7f1c780fc1c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:02.735 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c8023a700 1 -- 172.21.15.5:0/3762842992 >> 172.21.15.5:0/3762842992 conn(0x7f1c7806f440 msgr2=0x7f1c7806f870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:02.735 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c8023a700 1 -- 172.21.15.5:0/3762842992 shutdown_connections 2021-03-09T08:56:02.735 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c8023a700 1 -- 172.21.15.5:0/3762842992 wait complete. 2021-03-09T08:56:02.735 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c8023a700 1 Processor -- start 2021-03-09T08:56:02.736 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c8023a700 1 -- start start 2021-03-09T08:56:02.736 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c8023a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c780fbda0 0x7f1c7810a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:02.736 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c8023a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7810a870 con 0x7f1c780fbda0 2021-03-09T08:56:02.736 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c7dfd6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c780fbda0 0x7f1c7810a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:02.737 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c7dfd6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c780fbda0 0x7f1c7810a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46274/0 (socket says 172.21.15.5:46274) 2021-03-09T08:56:02.737 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c7dfd6700 1 -- 172.21.15.5:0/2683920097 learned_addr learned my addr 172.21.15.5:0/2683920097 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:02.737 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c7dfd6700 1 -- 172.21.15.5:0/2683920097 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7810a9b0 con 0x7f1c780fbda0 2021-03-09T08:56:02.737 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c7dfd6700 1 --2- 172.21.15.5:0/2683920097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c780fbda0 0x7f1c7810a380 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f1c74009420 tx=0x7f1c74006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:02.737 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c6a7fc700 1 -- 172.21.15.5:0/2683920097 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c74009ca0 con 0x7f1c780fbda0 2021-03-09T08:56:02.738 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c8023a700 1 -- 172.21.15.5:0/2683920097 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7810ac00 con 0x7f1c780fbda0 2021-03-09T08:56:02.739 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c8023a700 1 -- 172.21.15.5:0/2683920097 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c7810b760 con 0x7f1c780fbda0 2021-03-09T08:56:02.740 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c6a7fc700 1 -- 172.21.15.5:0/2683920097 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7f1c74011400 con 0x7f1c780fbda0 2021-03-09T08:56:02.740 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.729+0000 7f1c6a7fc700 1 -- 172.21.15.5:0/2683920097 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c7401bc50 con 0x7f1c780fbda0 2021-03-09T08:56:02.740 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.733+0000 7f1c6a7fc700 1 -- 172.21.15.5:0/2683920097 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 3) v1 ==== 35969+0+0 (secure 0 0 0) 0x7f1c7402b070 con 0x7f1c780fbda0 2021-03-09T08:56:02.740 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.733+0000 7f1c6a7fc700 1 --2- 172.21.15.5:0/2683920097 >> v2:172.21.15.5:6800/167667630 conn(0x7f1c6002fd30 0x7f1c600321f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:02.741 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.733+0000 7f1c6a7fc700 1 -- 172.21.15.5:0/2683920097 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1c740419b0 con 0x7f1c780fbda0 2021-03-09T08:56:02.741 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.733+0000 7f1c7d7d5700 1 --2- 172.21.15.5:0/2683920097 >> v2:172.21.15.5:6800/167667630 conn(0x7f1c6002fd30 0x7f1c600321f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:02.741 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.733+0000 7f1c8023a700 1 -- 172.21.15.5:0/2683920097 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c58004da0 con 0x7f1c780fbda0 2021-03-09T08:56:02.741 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.737+0000 7f1c7d7d5700 1 --2- 172.21.15.5:0/2683920097 >> v2:172.21.15.5:6800/167667630 conn(0x7f1c6002fd30 0x7f1c600321f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f1c6c0096a0 tx=0x7f1c6c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:02.742 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:02.741+0000 7f1c6a7fc700 1 -- 172.21.15.5:0/2683920097 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f1c74016020 con 0x7f1c780fbda0 2021-03-09T08:56:03.018 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:03.017+0000 7f1c6a7fc700 1 -- 172.21.15.5:0/2683920097 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mgrmap(e 4) v1 ==== 36049+0+0 (secure 0 0 0) 0x7f1c7400f070 con 0x7f1c780fbda0 2021-03-09T08:56:03.165 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:03.149+0000 7f1c8023a700 1 -- 172.21.15.5:0/2683920097 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1c58005050 con 0x7f1c780fbda0 2021-03-09T08:56:03.165 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:03.149+0000 7f1c6a7fc700 1 -- 172.21.15.5:0/2683920097 <== mon.0 v2:172.21.15.5:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1221 (secure 0 0 0) 0x7f1c7402c740 con 0x7f1c780fbda0 2021-03-09T08:56:03.165 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:03.166 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: { 2021-03-09T08:56:03.166 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "fsid": "252abbe0-80b5-11eb-907b-001a4aab830c", 2021-03-09T08:56:03.166 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "health": { 2021-03-09T08:56:03.167 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-03-09T08:56:03.167 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "checks": {}, 2021-03-09T08:56:03.167 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "mutes": [] 2021-03-09T08:56:03.167 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:56:03.167 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-03-09T08:56:03.168 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum": [ 2021-03-09T08:56:03.168 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 0 2021-03-09T08:56:03.168 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2021-03-09T08:56:03.168 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum_names": [ 2021-03-09T08:56:03.168 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "a" 2021-03-09T08:56:03.169 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2021-03-09T08:56:03.169 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum_age": 11, 2021-03-09T08:56:03.169 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "monmap": { 2021-03-09T08:56:03.169 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-09T08:56:03.170 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2021-03-09T08:56:03.170 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_mons": 1 2021-03-09T08:56:03.170 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:56:03.170 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osdmap": { 2021-03-09T08:56:03.170 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-09T08:56:03.171 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_osds": 0, 2021-03-09T08:56:03.171 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-03-09T08:56:03.171 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-03-09T08:56:03.171 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-03-09T08:56:03.171 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-03-09T08:56:03.172 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-03-09T08:56:03.172 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:56:03.172 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "pgmap": { 2021-03-09T08:56:03.172 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-03-09T08:56:03.173 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-03-09T08:56:03.173 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_pools": 0, 2021-03-09T08:56:03.173 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_objects": 0, 2021-03-09T08:56:03.173 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-03-09T08:56:03.173 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-03-09T08:56:03.174 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-03-09T08:56:03.174 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_total": 0 2021-03-09T08:56:03.174 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:56:03.174 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "fsmap": { 2021-03-09T08:56:03.174 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-09T08:56:03.175 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "by_rank": [], 2021-03-09T08:56:03.175 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "up:standby": 0 2021-03-09T08:56:03.175 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:56:03.175 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "mgrmap": { 2021-03-09T08:56:03.175 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "available": true, 2021-03-09T08:56:03.176 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-03-09T08:56:03.176 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "modules": [ 2021-03-09T08:56:03.176 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "iostat", 2021-03-09T08:56:03.176 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "restful" 2021-03-09T08:56:03.176 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2021-03-09T08:56:03.177 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "services": {} 2021-03-09T08:56:03.177 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:56:03.177 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "servicemap": { 2021-03-09T08:56:03.177 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-09T08:56:03.177 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "modified": "2021-03-09T08:55:44.624898+0000", 2021-03-09T08:56:03.178 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "services": {} 2021-03-09T08:56:03.178 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2021-03-09T08:56:03.178 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "progress_events": {} 2021-03-09T08:56:03.178 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: } 2021-03-09T08:56:03.179 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:03.153+0000 7f1c8023a700 1 -- 172.21.15.5:0/2683920097 >> v2:172.21.15.5:6800/167667630 conn(0x7f1c6002fd30 msgr2=0x7f1c600321f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:03.179 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:03.153+0000 7f1c8023a700 1 --2- 172.21.15.5:0/2683920097 >> v2:172.21.15.5:6800/167667630 conn(0x7f1c6002fd30 0x7f1c600321f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f1c6c0096a0 tx=0x7f1c6c006b40).stop 2021-03-09T08:56:03.179 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:03.153+0000 7f1c8023a700 1 -- 172.21.15.5:0/2683920097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c780fbda0 msgr2=0x7f1c7810a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:03.179 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:03.153+0000 7f1c8023a700 1 --2- 172.21.15.5:0/2683920097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c780fbda0 0x7f1c7810a380 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f1c74009420 tx=0x7f1c74006ee0).stop 2021-03-09T08:56:03.179 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:03.153+0000 7f1c8023a700 1 -- 172.21.15.5:0/2683920097 shutdown_connections 2021-03-09T08:56:03.180 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:03.153+0000 7f1c8023a700 1 --2- 172.21.15.5:0/2683920097 >> v2:172.21.15.5:6800/167667630 conn(0x7f1c6002fd30 0x7f1c600321f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:03.180 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:03.153+0000 7f1c8023a700 1 --2- 172.21.15.5:0/2683920097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c780fbda0 0x7f1c7810a380 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:03.180 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:03.153+0000 7f1c8023a700 1 -- 172.21.15.5:0/2683920097 >> 172.21.15.5:0/2683920097 conn(0x7f1c7806f440 msgr2=0x7f1c780f97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:03.180 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:03.153+0000 7f1c8023a700 1 -- 172.21.15.5:0/2683920097 shutdown_connections 2021-03-09T08:56:03.180 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:03.153+0000 7f1c8023a700 1 -- 172.21.15.5:0/2683920097 wait complete. 2021-03-09T08:56:03.875 INFO:teuthology.orchestra.run.smithi005.stderr:mgr is available 2021-03-09T08:56:03.875 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 config-key set mgr/cephadm/ssh_user root 2021-03-09T08:56:04.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:04 smithi005 bash[12738]: cluster 2021-03-09T08:56:03.023332+0000 mon.a (mon.0) 23 : cluster [DBG] mgrmap e4: y(active, since 2s) 2021-03-09T08:56:04.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:04 smithi005 bash[12738]: audit 2021-03-09T08:56:03.155198+0000 mon.a (mon.0) 24 : audit [DBG] from='client.? 172.21.15.5:0/2683920097' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-03-09T08:56:05.023 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.021+0000 7ff3934f1700 1 Processor -- start 2021-03-09T08:56:05.023 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.021+0000 7ff3934f1700 1 -- start start 2021-03-09T08:56:05.023 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.021+0000 7ff3934f1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff38c0f7fc0 0x7ff38c0f83e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:05.023 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.021+0000 7ff3934f1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c0f88d0 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.025 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.021+0000 7ff3924ef700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff38c0f7fc0 0x7ff38c0f83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:05.025 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.021+0000 7ff3924ef700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff38c0f7fc0 0x7ff38c0f83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46278/0 (socket says 172.21.15.5:46278) 2021-03-09T08:56:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.021+0000 7ff3924ef700 1 -- 172.21.15.5:0/2078680072 learned_addr learned my addr 172.21.15.5:0/2078680072 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.021+0000 7ff3924ef700 1 -- 172.21.15.5:0/2078680072 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c0f8a10 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.021+0000 7ff3924ef700 1 --2- 172.21.15.5:0/2078680072 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff38c0f7fc0 0x7ff38c0f83e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7ff38800b950 tx=0x7ff388006ee0).ready entity=mon.0 client_cookie=7b0b1876bebbbddb server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.021+0000 7ff3914ed700 1 -- 172.21.15.5:0/2078680072 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff388009840 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.027 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.025+0000 7ff3914ed700 1 -- 172.21.15.5:0/2078680072 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7ff388009e00 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.027 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.025+0000 7ff3914ed700 1 -- 172.21.15.5:0/2078680072 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff388011c80 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.027 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.025+0000 7ff3934f1700 1 -- 172.21.15.5:0/2078680072 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff38c0f7fc0 msgr2=0x7ff38c0f83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:05.031 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.025+0000 7ff3934f1700 1 --2- 172.21.15.5:0/2078680072 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff38c0f7fc0 0x7ff38c0f83e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7ff38800b950 tx=0x7ff388006ee0).stop 2021-03-09T08:56:05.031 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.025+0000 7ff3934f1700 1 -- 172.21.15.5:0/2078680072 shutdown_connections 2021-03-09T08:56:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.025+0000 7ff3934f1700 1 --2- 172.21.15.5:0/2078680072 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff38c0f7fc0 0x7ff38c0f83e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.025+0000 7ff3934f1700 1 -- 172.21.15.5:0/2078680072 >> 172.21.15.5:0/2078680072 conn(0x7ff38c0f37e0 msgr2=0x7ff38c0f5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.025+0000 7ff3934f1700 1 -- 172.21.15.5:0/2078680072 shutdown_connections 2021-03-09T08:56:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.025+0000 7ff3934f1700 1 -- 172.21.15.5:0/2078680072 wait complete. 2021-03-09T08:56:05.033 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.025+0000 7ff3934f1700 1 Processor -- start 2021-03-09T08:56:05.033 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.025+0000 7ff3934f1700 1 -- start start 2021-03-09T08:56:05.034 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.025+0000 7ff3934f1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff38c0f7fc0 0x7ff38c1076c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:05.034 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.025+0000 7ff3934f1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c105b40 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.034 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.025+0000 7ff3924ef700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff38c0f7fc0 0x7ff38c1076c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.025+0000 7ff3924ef700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff38c0f7fc0 0x7ff38c1076c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46280/0 (socket says 172.21.15.5:46280) 2021-03-09T08:56:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.025+0000 7ff3924ef700 1 -- 172.21.15.5:0/2169651993 learned_addr learned my addr 172.21.15.5:0/2169651993 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.029+0000 7ff3924ef700 1 -- 172.21.15.5:0/2169651993 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c105c80 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.029+0000 7ff3924ef700 1 --2- 172.21.15.5:0/2169651993 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff38c0f7fc0 0x7ff38c1076c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7ff388009420 tx=0x7ff388006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.029+0000 7ff37affd700 1 -- 172.21.15.5:0/2169651993 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff388009ca0 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.036 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.029+0000 7ff37affd700 1 -- 172.21.15.5:0/2169651993 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7ff388011400 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.036 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.029+0000 7ff3934f1700 1 -- 172.21.15.5:0/2169651993 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38c105e70 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.036 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.029+0000 7ff37affd700 1 -- 172.21.15.5:0/2169651993 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff38801bb50 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.036 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.029+0000 7ff3934f1700 1 -- 172.21.15.5:0/2169651993 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38c106300 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.037 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.029+0000 7ff37affd700 1 -- 172.21.15.5:0/2169651993 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 4) v1 ==== 36049+0+0 (secure 0 0 0) 0x7ff38802b070 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.038 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.029+0000 7ff378ff9700 1 -- 172.21.15.5:0/2169651993 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff370004da0 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.039 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.029+0000 7ff37affd700 1 --2- 172.21.15.5:0/2169651993 >> v2:172.21.15.5:6800/167667630 conn(0x7ff37402fdc0 0x7ff374032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:05.039 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.029+0000 7ff37affd700 1 -- 172.21.15.5:0/2169651993 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff388041c80 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.040 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.029+0000 7ff391cee700 1 --2- 172.21.15.5:0/2169651993 >> v2:172.21.15.5:6800/167667630 conn(0x7ff37402fdc0 0x7ff374032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:05.040 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.029+0000 7ff391cee700 1 --2- 172.21.15.5:0/2169651993 >> v2:172.21.15.5:6800/167667630 conn(0x7ff37402fdc0 0x7ff374032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff37c0096a0 tx=0x7ff37c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:05.040 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.037+0000 7ff37affd700 1 -- 172.21.15.5:0/2169651993 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7ff388022960 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.497 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.497+0000 7ff378ff9700 1 -- 172.21.15.5:0/2169651993 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command([{prefix=config-key set, key=mgr/cephadm/ssh_user}] v 0) v1 -- 0x7ff3700055d0 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.502 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.501+0000 7ff37affd700 1 -- 172.21.15.5:0/2169651993 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/cephadm/ssh_user}]=0 set mgr/cephadm/ssh_user v7)=0 set mgr/cephadm/ssh_user v7) v1 ==== 132+0+0 (secure 0 0 0) 0x7ff388016030 con 0x7ff38c0f7fc0 2021-03-09T08:56:05.504 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.501+0000 7ff378ff9700 1 -- 172.21.15.5:0/2169651993 >> v2:172.21.15.5:6800/167667630 conn(0x7ff37402fdc0 msgr2=0x7ff374032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:05.505 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.501+0000 7ff378ff9700 1 --2- 172.21.15.5:0/2169651993 >> v2:172.21.15.5:6800/167667630 conn(0x7ff37402fdc0 0x7ff374032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff37c0096a0 tx=0x7ff37c006b40).stop 2021-03-09T08:56:05.505 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.501+0000 7ff378ff9700 1 -- 172.21.15.5:0/2169651993 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff38c0f7fc0 msgr2=0x7ff38c1076c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:05.505 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.501+0000 7ff378ff9700 1 --2- 172.21.15.5:0/2169651993 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff38c0f7fc0 0x7ff38c1076c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7ff388009420 tx=0x7ff388006ee0).stop 2021-03-09T08:56:05.506 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.505+0000 7ff378ff9700 1 -- 172.21.15.5:0/2169651993 shutdown_connections 2021-03-09T08:56:05.507 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.505+0000 7ff378ff9700 1 --2- 172.21.15.5:0/2169651993 >> v2:172.21.15.5:6800/167667630 conn(0x7ff37402fdc0 0x7ff374032280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:05.507 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.505+0000 7ff378ff9700 1 --2- 172.21.15.5:0/2169651993 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff38c0f7fc0 0x7ff38c1076c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:05.507 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.505+0000 7ff378ff9700 1 -- 172.21.15.5:0/2169651993 >> 172.21.15.5:0/2169651993 conn(0x7ff38c0f37e0 msgr2=0x7ff38c0fcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:05.507 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.505+0000 7ff378ff9700 1 -- 172.21.15.5:0/2169651993 shutdown_connections 2021-03-09T08:56:05.508 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:05.505+0000 7ff378ff9700 1 -- 172.21.15.5:0/2169651993 wait complete. 2021-03-09T08:56:05.508 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: set mgr/cephadm/ssh_user 2021-03-09T08:56:06.132 INFO:teuthology.orchestra.run.smithi005.stderr:Enabling cephadm module... 2021-03-09T08:56:06.133 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 mgr module enable cephadm 2021-03-09T08:56:06.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:06 smithi005 bash[12738]: audit 2021-03-09T08:56:05.505472+0000 mon.a (mon.0) 25 : audit [INF] from='client.? 172.21.15.5:0/2169651993' entity='client.admin' 2021-03-09T08:56:07.250 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.249+0000 7f053bfff700 1 Processor -- start 2021-03-09T08:56:07.250 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.249+0000 7f053bfff700 1 -- start start 2021-03-09T08:56:07.250 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.249+0000 7f053bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f053c06fcf0 0x7f053c06e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:07.251 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.249+0000 7f053bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c070110 con 0x7f053c06fcf0 2021-03-09T08:56:07.251 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.249+0000 7f053affd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f053c06fcf0 0x7f053c06e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:07.253 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.249+0000 7f053affd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f053c06fcf0 0x7f053c06e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46284/0 (socket says 172.21.15.5:46284) 2021-03-09T08:56:07.253 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.249+0000 7f053affd700 1 -- 172.21.15.5:0/2214228910 learned_addr learned my addr 172.21.15.5:0/2214228910 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:07.253 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.249+0000 7f053affd700 1 -- 172.21.15.5:0/2214228910 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c06e780 con 0x7f053c06fcf0 2021-03-09T08:56:07.254 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.249+0000 7f053affd700 1 --2- 172.21.15.5:0/2214228910 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f053c06fcf0 0x7f053c06e290 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f052c00b950 tx=0x7f052c006ee0).ready entity=mon.0 client_cookie=7be78e87cee092a7 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:07.254 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.249+0000 7f053a7fc700 1 -- 172.21.15.5:0/2214228910 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f052c009840 con 0x7f053c06fcf0 2021-03-09T08:56:07.254 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.249+0000 7f053a7fc700 1 -- 172.21.15.5:0/2214228910 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7f052c009e00 con 0x7f053c06fcf0 2021-03-09T08:56:07.254 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.249+0000 7f053a7fc700 1 -- 172.21.15.5:0/2214228910 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f052c011c80 con 0x7f053c06fcf0 2021-03-09T08:56:07.254 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.253+0000 7f053bfff700 1 -- 172.21.15.5:0/2214228910 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f053c06fcf0 msgr2=0x7f053c06e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:07.255 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.253+0000 7f053bfff700 1 --2- 172.21.15.5:0/2214228910 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f053c06fcf0 0x7f053c06e290 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f052c00b950 tx=0x7f052c006ee0).stop 2021-03-09T08:56:07.255 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.253+0000 7f053bfff700 1 -- 172.21.15.5:0/2214228910 shutdown_connections 2021-03-09T08:56:07.255 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.253+0000 7f053bfff700 1 --2- 172.21.15.5:0/2214228910 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f053c06fcf0 0x7f053c06e290 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:07.255 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.253+0000 7f053bfff700 1 -- 172.21.15.5:0/2214228910 >> 172.21.15.5:0/2214228910 conn(0x7f053c0f5120 msgr2=0x7f053c0f7560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:07.256 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.253+0000 7f053bfff700 1 -- 172.21.15.5:0/2214228910 shutdown_connections 2021-03-09T08:56:07.256 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.253+0000 7f053bfff700 1 -- 172.21.15.5:0/2214228910 wait complete. 2021-03-09T08:56:07.259 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.253+0000 7f053bfff700 1 Processor -- start 2021-03-09T08:56:07.259 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.253+0000 7f053bfff700 1 -- start start 2021-03-09T08:56:07.259 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.253+0000 7f053bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f053c06fcf0 0x7f053c10a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:07.260 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.253+0000 7f053bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c10a780 con 0x7f053c06fcf0 2021-03-09T08:56:07.260 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.253+0000 7f053affd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f053c06fcf0 0x7f053c10a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:07.260 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.253+0000 7f053affd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f053c06fcf0 0x7f053c10a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46286/0 (socket says 172.21.15.5:46286) 2021-03-09T08:56:07.261 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.253+0000 7f053affd700 1 -- 172.21.15.5:0/1427797405 learned_addr learned my addr 172.21.15.5:0/1427797405 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:07.261 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.253+0000 7f053affd700 1 -- 172.21.15.5:0/1427797405 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c10a8c0 con 0x7f053c06fcf0 2021-03-09T08:56:07.261 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.253+0000 7f053affd700 1 --2- 172.21.15.5:0/1427797405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f053c06fcf0 0x7f053c10a290 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f052c009420 tx=0x7f052c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:07.261 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.257+0000 7f05337fe700 1 -- 172.21.15.5:0/1427797405 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f052c009ca0 con 0x7f053c06fcf0 2021-03-09T08:56:07.262 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.257+0000 7f05337fe700 1 -- 172.21.15.5:0/1427797405 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7f052c010f00 con 0x7f053c06fcf0 2021-03-09T08:56:07.262 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.257+0000 7f05337fe700 1 -- 172.21.15.5:0/1427797405 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f052c01b9f0 con 0x7f053c06fcf0 2021-03-09T08:56:07.262 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.257+0000 7f053bfff700 1 -- 172.21.15.5:0/1427797405 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053c10ab10 con 0x7f053c06fcf0 2021-03-09T08:56:07.262 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.257+0000 7f053bfff700 1 -- 172.21.15.5:0/1427797405 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053c10b670 con 0x7f053c06fcf0 2021-03-09T08:56:07.262 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.257+0000 7f05337fe700 1 -- 172.21.15.5:0/1427797405 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 4) v1 ==== 36049+0+0 (secure 0 0 0) 0x7f052c02b070 con 0x7f053c06fcf0 2021-03-09T08:56:07.263 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.257+0000 7f05337fe700 1 --2- 172.21.15.5:0/1427797405 >> v2:172.21.15.5:6800/167667630 conn(0x7f051c02fde0 0x7f051c0322a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:07.263 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.257+0000 7f05337fe700 1 -- 172.21.15.5:0/1427797405 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f052c0426e0 con 0x7f053c06fcf0 2021-03-09T08:56:07.264 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.257+0000 7f0533fff700 1 --2- 172.21.15.5:0/1427797405 >> v2:172.21.15.5:6800/167667630 conn(0x7f051c02fde0 0x7f051c0322a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:07.264 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.257+0000 7f0533fff700 1 --2- 172.21.15.5:0/1427797405 >> v2:172.21.15.5:6800/167667630 conn(0x7f051c02fde0 0x7f051c0322a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f05240096a0 tx=0x7f0524006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:07.264 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.257+0000 7f053bfff700 1 -- 172.21.15.5:0/1427797405 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053c0589f0 con 0x7f053c06fcf0 2021-03-09T08:56:07.268 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.269+0000 7f05337fe700 1 -- 172.21.15.5:0/1427797405 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f052c016070 con 0x7f053c06fcf0 2021-03-09T08:56:07.637 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.637+0000 7f053bfff700 1 -- 172.21.15.5:0/1427797405 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f053c0ffbe0 con 0x7f053c06fcf0 2021-03-09T08:56:07.848 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.845+0000 7f05337fe700 1 -- 172.21.15.5:0/1427797405 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mgrmap(e 5) v1 ==== 36060+0+0 (secure 0 0 0) 0x7f052c00f070 con 0x7f053c06fcf0 2021-03-09T08:56:07.849 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.849+0000 7f05337fe700 1 -- 172.21.15.5:0/1427797405 <== mon.0 v2:172.21.15.5:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f052c041ad0 con 0x7f053c06fcf0 2021-03-09T08:56:07.856 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.853+0000 7f053bfff700 1 -- 172.21.15.5:0/1427797405 >> v2:172.21.15.5:6800/167667630 conn(0x7f051c02fde0 msgr2=0x7f051c0322a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:07.856 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.853+0000 7f053bfff700 1 --2- 172.21.15.5:0/1427797405 >> v2:172.21.15.5:6800/167667630 conn(0x7f051c02fde0 0x7f051c0322a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f05240096a0 tx=0x7f0524006b40).stop 2021-03-09T08:56:07.857 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.853+0000 7f053bfff700 1 -- 172.21.15.5:0/1427797405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f053c06fcf0 msgr2=0x7f053c10a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:07.857 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.853+0000 7f053bfff700 1 --2- 172.21.15.5:0/1427797405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f053c06fcf0 0x7f053c10a290 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f052c009420 tx=0x7f052c006ee0).stop 2021-03-09T08:56:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.853+0000 7f053bfff700 1 -- 172.21.15.5:0/1427797405 shutdown_connections 2021-03-09T08:56:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.853+0000 7f053bfff700 1 --2- 172.21.15.5:0/1427797405 >> v2:172.21.15.5:6800/167667630 conn(0x7f051c02fde0 0x7f051c0322a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.853+0000 7f053bfff700 1 --2- 172.21.15.5:0/1427797405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f053c06fcf0 0x7f053c10a290 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:07.859 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.853+0000 7f053bfff700 1 -- 172.21.15.5:0/1427797405 >> 172.21.15.5:0/1427797405 conn(0x7f053c0f5120 msgr2=0x7f053c0f5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:07.859 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.853+0000 7f053bfff700 1 -- 172.21.15.5:0/1427797405 shutdown_connections 2021-03-09T08:56:07.860 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:07.853+0000 7f053bfff700 1 -- 172.21.15.5:0/1427797405 wait complete. 2021-03-09T08:56:08.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:07 smithi005 bash[12738]: audit 2021-03-09T08:56:07.641655+0000 mon.a (mon.0) 26 : audit [INF] from='client.? 172.21.15.5:0/1427797405' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2021-03-09T08:56:08.114 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:07 smithi005 bash[12977]: ignoring --setgroup ceph since I am not root 2021-03-09T08:56:08.527 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 mgr stat 2021-03-09T08:56:09.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:08 smithi005 bash[12738]: audit 2021-03-09T08:56:07.853207+0000 mon.a (mon.0) 27 : audit [INF] from='client.? 172.21.15.5:0/1427797405' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2021-03-09T08:56:09.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:08 smithi005 bash[12738]: cluster 2021-03-09T08:56:07.853328+0000 mon.a (mon.0) 28 : cluster [DBG] mgrmap e5: y(active, since 6s) 2021-03-09T08:56:09.668 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.665+0000 7f30416ab700 1 Processor -- start 2021-03-09T08:56:09.668 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.665+0000 7f30416ab700 1 -- start start 2021-03-09T08:56:09.670 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.665+0000 7f30416ab700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f303c06dc10 0x7f303c0fc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:09.670 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.665+0000 7f30416ab700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c0fcca0 con 0x7f303c06dc10 2021-03-09T08:56:09.670 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.665+0000 7f303affd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f303c06dc10 0x7f303c0fc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:09.671 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.665+0000 7f303affd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f303c06dc10 0x7f303c0fc7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46302/0 (socket says 172.21.15.5:46302) 2021-03-09T08:56:09.671 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.665+0000 7f303affd700 1 -- 172.21.15.5:0/462102554 learned_addr learned my addr 172.21.15.5:0/462102554 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:09.671 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f303affd700 1 -- 172.21.15.5:0/462102554 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303c0fcde0 con 0x7f303c06dc10 2021-03-09T08:56:09.671 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f303affd700 1 --2- 172.21.15.5:0/462102554 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f303c06dc10 0x7f303c0fc7b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f302c00b950 tx=0x7f302c006ee0).ready entity=mon.0 client_cookie=672f5f6d9188503d server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:09.672 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f3039ffb700 1 -- 172.21.15.5:0/462102554 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f302c0098d0 con 0x7f303c06dc10 2021-03-09T08:56:09.672 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f3039ffb700 1 -- 172.21.15.5:0/462102554 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7f302c009e90 con 0x7f303c06dc10 2021-03-09T08:56:09.672 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f3039ffb700 1 -- 172.21.15.5:0/462102554 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f302c011d10 con 0x7f303c06dc10 2021-03-09T08:56:09.672 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f30416ab700 1 -- 172.21.15.5:0/462102554 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f303c06dc10 msgr2=0x7f303c0fc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:09.673 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f30416ab700 1 --2- 172.21.15.5:0/462102554 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f303c06dc10 0x7f303c0fc7b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f302c00b950 tx=0x7f302c006ee0).stop 2021-03-09T08:56:09.676 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f30416ab700 1 -- 172.21.15.5:0/462102554 shutdown_connections 2021-03-09T08:56:09.676 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f30416ab700 1 --2- 172.21.15.5:0/462102554 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f303c06dc10 0x7f303c0fc7b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:09.677 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f30416ab700 1 -- 172.21.15.5:0/462102554 >> 172.21.15.5:0/462102554 conn(0x7f303c0f41f0 msgr2=0x7f303c0f6630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:09.677 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f30416ab700 1 -- 172.21.15.5:0/462102554 shutdown_connections 2021-03-09T08:56:09.677 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f30416ab700 1 -- 172.21.15.5:0/462102554 wait complete. 2021-03-09T08:56:09.677 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f30416ab700 1 Processor -- start 2021-03-09T08:56:09.678 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f30416ab700 1 -- start start 2021-03-09T08:56:09.678 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f30416ab700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f303c06dc10 0x7f303c1081f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:09.678 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f30416ab700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c1086e0 con 0x7f303c06dc10 2021-03-09T08:56:09.678 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f303affd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f303c06dc10 0x7f303c1081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:09.678 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f303affd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f303c06dc10 0x7f303c1081f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46304/0 (socket says 172.21.15.5:46304) 2021-03-09T08:56:09.679 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.669+0000 7f303affd700 1 -- 172.21.15.5:0/814859561 learned_addr learned my addr 172.21.15.5:0/814859561 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:09.679 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.673+0000 7f303affd700 1 -- 172.21.15.5:0/814859561 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303c108820 con 0x7f303c06dc10 2021-03-09T08:56:09.679 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.673+0000 7f303affd700 1 --2- 172.21.15.5:0/814859561 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f303c06dc10 0x7f303c1081f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f302c009420 tx=0x7f302c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:09.679 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.673+0000 7f30237fe700 1 -- 172.21.15.5:0/814859561 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f302c011b50 con 0x7f303c06dc10 2021-03-09T08:56:09.680 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.673+0000 7f30237fe700 1 -- 172.21.15.5:0/814859561 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7f302c011770 con 0x7f303c06dc10 2021-03-09T08:56:09.680 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.673+0000 7f30237fe700 1 -- 172.21.15.5:0/814859561 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f302c01bb00 con 0x7f303c06dc10 2021-03-09T08:56:09.680 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.673+0000 7f30416ab700 1 -- 172.21.15.5:0/814859561 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303c108a70 con 0x7f303c06dc10 2021-03-09T08:56:09.680 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.673+0000 7f30416ab700 1 -- 172.21.15.5:0/814859561 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303c1095d0 con 0x7f303c06dc10 2021-03-09T08:56:09.680 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.673+0000 7f30237fe700 1 -- 172.21.15.5:0/814859561 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 36060+0+0 (secure 0 0 0) 0x7f302c02b070 con 0x7f303c06dc10 2021-03-09T08:56:09.681 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.673+0000 7f30237fe700 1 --2- 172.21.15.5:0/814859561 >> v2:172.21.15.5:6800/167667630 conn(0x7f302802fe20 0x7f30280322e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:09.681 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.673+0000 7f30237fe700 1 -- 172.21.15.5:0/814859561 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f302c0427e0 con 0x7f303c06dc10 2021-03-09T08:56:09.681 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.673+0000 7f303a7fc700 1 -- 172.21.15.5:0/814859561 >> v2:172.21.15.5:6800/167667630 conn(0x7f302802fe20 msgr2=0x7f30280322e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:6800/167667630 2021-03-09T08:56:09.681 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.673+0000 7f303a7fc700 1 --2- 172.21.15.5:0/814859561 >> v2:172.21.15.5:6800/167667630 conn(0x7f302802fe20 0x7f30280322e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:09.681 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.673+0000 7f30416ab700 1 -- 172.21.15.5:0/814859561 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303c0589f0 con 0x7f303c06dc10 2021-03-09T08:56:09.682 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.681+0000 7f30237fe700 1 -- 172.21.15.5:0/814859561 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f302c016030 con 0x7f303c06dc10 2021-03-09T08:56:09.876 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.877+0000 7f303a7fc700 1 -- 172.21.15.5:0/814859561 >> v2:172.21.15.5:6800/167667630 conn(0x7f302802fe20 msgr2=0x7f30280322e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:6800/167667630 2021-03-09T08:56:09.877 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:09.877+0000 7f303a7fc700 1 --2- 172.21.15.5:0/814859561 >> v2:172.21.15.5:6800/167667630 conn(0x7f302802fe20 0x7f30280322e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T08:56:10.047 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:10.045+0000 7f30416ab700 1 -- 172.21.15.5:0/814859561 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f303c0fdab0 con 0x7f303c06dc10 2021-03-09T08:56:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:10.045+0000 7f30237fe700 1 -- 172.21.15.5:0/814859561 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f302c024400 con 0x7f303c06dc10 2021-03-09T08:56:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: { 2021-03-09T08:56:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 5, 2021-03-09T08:56:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "available": true, 2021-03-09T08:56:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "active_name": "y", 2021-03-09T08:56:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_standby": 0 2021-03-09T08:56:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: } 2021-03-09T08:56:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:10.049+0000 7f30416ab700 1 -- 172.21.15.5:0/814859561 >> v2:172.21.15.5:6800/167667630 conn(0x7f302802fe20 msgr2=0x7f30280322e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:10.049+0000 7f30416ab700 1 --2- 172.21.15.5:0/814859561 >> v2:172.21.15.5:6800/167667630 conn(0x7f302802fe20 0x7f30280322e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:10.049+0000 7f30416ab700 1 -- 172.21.15.5:0/814859561 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f303c06dc10 msgr2=0x7f303c1081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:10.049+0000 7f30416ab700 1 --2- 172.21.15.5:0/814859561 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f303c06dc10 0x7f303c1081f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f302c009420 tx=0x7f302c006ee0).stop 2021-03-09T08:56:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:10.049+0000 7f30416ab700 1 -- 172.21.15.5:0/814859561 shutdown_connections 2021-03-09T08:56:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:10.049+0000 7f30416ab700 1 --2- 172.21.15.5:0/814859561 >> v2:172.21.15.5:6800/167667630 conn(0x7f302802fe20 0x7f30280322e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:10.049+0000 7f30416ab700 1 --2- 172.21.15.5:0/814859561 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f303c06dc10 0x7f303c1081f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:10.049+0000 7f30416ab700 1 -- 172.21.15.5:0/814859561 >> 172.21.15.5:0/814859561 conn(0x7f303c0f41f0 msgr2=0x7f303c0f4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:10.049+0000 7f30416ab700 1 -- 172.21.15.5:0/814859561 shutdown_connections 2021-03-09T08:56:10.053 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:10.049+0000 7f30416ab700 1 -- 172.21.15.5:0/814859561 wait complete. 2021-03-09T08:56:10.363 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:10 smithi005 bash[12738]: audit 2021-03-09T08:56:10.050406+0000 mon.a (mon.0) 29 : audit [DBG] from='client.? 172.21.15.5:0/814859561' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-03-09T08:56:10.751 INFO:teuthology.orchestra.run.smithi005.stderr:Waiting for the mgr to restart... 2021-03-09T08:56:10.752 INFO:teuthology.orchestra.run.smithi005.stderr:Waiting for mgr epoch 5... 2021-03-09T08:56:10.752 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 tell mgr mgr_status 2021-03-09T08:56:11.900 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.897+0000 7fa38ccc2700 1 Processor -- start 2021-03-09T08:56:11.900 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.897+0000 7fa38ccc2700 1 -- start start 2021-03-09T08:56:11.901 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38ccc2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3880f9b20 0x7fa3880f9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:11.901 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38ccc2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3880fa430 con 0x7fa3880f9b20 2021-03-09T08:56:11.902 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38659c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3880f9b20 0x7fa3880f9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:11.902 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38659c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3880f9b20 0x7fa3880f9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46312/0 (socket says 172.21.15.5:46312) 2021-03-09T08:56:11.902 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38659c700 1 -- 172.21.15.5:0/3887511459 learned_addr learned my addr 172.21.15.5:0/3887511459 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:11.902 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38659c700 1 -- 172.21.15.5:0/3887511459 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3880fa570 con 0x7fa3880f9b20 2021-03-09T08:56:11.903 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38659c700 1 --2- 172.21.15.5:0/3887511459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3880f9b20 0x7fa3880f9f40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fa37800b950 tx=0x7fa378006ee0).ready entity=mon.0 client_cookie=af1b22ed80ba61f5 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:11.903 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38559a700 1 -- 172.21.15.5:0/3887511459 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa378009840 con 0x7fa3880f9b20 2021-03-09T08:56:11.903 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38559a700 1 -- 172.21.15.5:0/3887511459 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7fa378009e00 con 0x7fa3880f9b20 2021-03-09T08:56:11.903 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38559a700 1 -- 172.21.15.5:0/3887511459 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa378011c80 con 0x7fa3880f9b20 2021-03-09T08:56:11.906 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38ccc2700 1 -- 172.21.15.5:0/3887511459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3880f9b20 msgr2=0x7fa3880f9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:11.906 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38ccc2700 1 --2- 172.21.15.5:0/3887511459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3880f9b20 0x7fa3880f9f40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fa37800b950 tx=0x7fa378006ee0).stop 2021-03-09T08:56:11.906 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38ccc2700 1 -- 172.21.15.5:0/3887511459 shutdown_connections 2021-03-09T08:56:11.907 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38ccc2700 1 --2- 172.21.15.5:0/3887511459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3880f9b20 0x7fa3880f9f40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:11.907 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38ccc2700 1 -- 172.21.15.5:0/3887511459 >> 172.21.15.5:0/3887511459 conn(0x7fa3880f5300 msgr2=0x7fa3880f7740 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:11.907 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38ccc2700 1 -- 172.21.15.5:0/3887511459 shutdown_connections 2021-03-09T08:56:11.907 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.901+0000 7fa38ccc2700 1 -- 172.21.15.5:0/3887511459 wait complete. 2021-03-09T08:56:11.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa38ccc2700 1 Processor -- start 2021-03-09T08:56:11.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa38ccc2700 1 -- start start 2021-03-09T08:56:11.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa38ccc2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3880f9b20 0x7fa388108270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:11.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa38ccc2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa388108760 con 0x7fa3880f9b20 2021-03-09T08:56:11.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa38659c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3880f9b20 0x7fa388108270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:11.909 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa38659c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3880f9b20 0x7fa388108270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46314/0 (socket says 172.21.15.5:46314) 2021-03-09T08:56:11.909 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa38659c700 1 -- 172.21.15.5:0/3629699819 learned_addr learned my addr 172.21.15.5:0/3629699819 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:11.909 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa38659c700 1 -- 172.21.15.5:0/3629699819 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3881088a0 con 0x7fa3880f9b20 2021-03-09T08:56:11.910 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa38659c700 1 --2- 172.21.15.5:0/3629699819 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3880f9b20 0x7fa388108270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fa378009420 tx=0x7fa378006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:11.910 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa376ffd700 1 -- 172.21.15.5:0/3629699819 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa378009ca0 con 0x7fa3880f9b20 2021-03-09T08:56:11.911 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa376ffd700 1 -- 172.21.15.5:0/3629699819 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7fa378011400 con 0x7fa3880f9b20 2021-03-09T08:56:11.911 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa376ffd700 1 -- 172.21.15.5:0/3629699819 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa37801bb50 con 0x7fa3880f9b20 2021-03-09T08:56:11.911 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa38ccc2700 1 -- 172.21.15.5:0/3629699819 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa388108af0 con 0x7fa3880f9b20 2021-03-09T08:56:11.911 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa38ccc2700 1 -- 172.21.15.5:0/3629699819 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa388109650 con 0x7fa3880f9b20 2021-03-09T08:56:11.912 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa376ffd700 1 -- 172.21.15.5:0/3629699819 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 36060+0+0 (secure 0 0 0) 0x7fa37802b070 con 0x7fa3880f9b20 2021-03-09T08:56:11.912 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa376ffd700 1 --2- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/167667630 conn(0x7fa36c02fe20 0x7fa36c0322e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:11.912 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.905+0000 7fa376ffd700 1 -- 172.21.15.5:0/3629699819 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa378041b10 con 0x7fa3880f9b20 2021-03-09T08:56:11.912 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.909+0000 7fa385d9b700 1 -- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/167667630 conn(0x7fa36c02fe20 msgr2=0x7fa36c0322e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:6800/167667630 2021-03-09T08:56:11.912 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.909+0000 7fa385d9b700 1 --2- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/167667630 conn(0x7fa36c02fe20 0x7fa36c0322e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:11.913 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:11.909+0000 7fa38ccc2700 1 -- 172.21.15.5:0/3629699819 --> v2:172.21.15.5:6800/167667630 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3880589f0 con 0x7fa36c02fe20 2021-03-09T08:56:12.112 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:12.113+0000 7fa385d9b700 1 -- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/167667630 conn(0x7fa36c02fe20 msgr2=0x7fa36c0322e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:6800/167667630 2021-03-09T08:56:12.113 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:12.113+0000 7fa385d9b700 1 --2- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/167667630 conn(0x7fa36c02fe20 0x7fa36c0322e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T08:56:12.516 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:12.517+0000 7fa385d9b700 1 -- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/167667630 conn(0x7fa36c02fe20 msgr2=0x7fa36c0322e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:6800/167667630 2021-03-09T08:56:12.641 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:12.517+0000 7fa385d9b700 1 --2- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/167667630 conn(0x7fa36c02fe20 0x7fa36c0322e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-03-09T08:56:13.320 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:13.321+0000 7fa385d9b700 1 -- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/167667630 conn(0x7fa36c02fe20 msgr2=0x7fa36c0322e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:6800/167667630 2021-03-09T08:56:13.321 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:13.321+0000 7fa385d9b700 1 --2- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/167667630 conn(0x7fa36c02fe20 0x7fa36c0322e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-03-09T08:56:14.265 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:14.265+0000 7fa376ffd700 1 -- 172.21.15.5:0/3629699819 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mgrmap(e 6) v1 ==== 35945+0+0 (secure 0 0 0) 0x7fa37800f070 con 0x7fa3880f9b20 2021-03-09T08:56:14.265 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:14.265+0000 7fa376ffd700 1 -- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/167667630 conn(0x7fa36c02fe20 msgr2=0x7fa36c0322e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:14.266 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:14.265+0000 7fa376ffd700 1 --2- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/167667630 conn(0x7fa36c02fe20 0x7fa36c0322e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:15.266 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.265+0000 7fa376ffd700 1 -- 172.21.15.5:0/3629699819 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mgrmap(e 7) v1 ==== 35980+0+0 (secure 0 0 0) 0x7fa378010db0 con 0x7fa3880f9b20 2021-03-09T08:56:15.267 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.265+0000 7fa376ffd700 1 --2- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/4233713404 conn(0x7fa36c033690 0x7fa36c035a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:15.267 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.265+0000 7fa376ffd700 1 -- 172.21.15.5:0/3629699819 --> v2:172.21.15.5:6800/4233713404 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3880589f0 con 0x7fa36c033690 2021-03-09T08:56:15.268 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.265+0000 7fa385d9b700 1 --2- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/4233713404 conn(0x7fa36c033690 0x7fa36c035a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:15.269 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.269+0000 7fa385d9b700 1 --2- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/4233713404 conn(0x7fa36c033690 0x7fa36c035a80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fa37c003720 tx=0x7fa37c006d10).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:15.270 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.269+0000 7fa376ffd700 1 -- 172.21.15.5:0/3629699819 <== mgr.14116 v2:172.21.15.5:6800/4233713404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fa3880589f0 con 0x7fa36c033690 2021-03-09T08:56:15.286 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.285+0000 7fa38ccc2700 1 -- 172.21.15.5:0/3629699819 --> v2:172.21.15.5:6800/4233713404 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa388045280 con 0x7fa36c033690 2021-03-09T08:56:15.287 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.285+0000 7fa376ffd700 1 -- 172.21.15.5:0/3629699819 <== mgr.14116 v2:172.21.15.5:6800/4233713404 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa388045280 con 0x7fa36c033690 2021-03-09T08:56:15.287 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: { 2021-03-09T08:56:15.287 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2021-03-09T08:56:15.287 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "initialized": true 2021-03-09T08:56:15.289 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: } 2021-03-09T08:56:15.289 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.285+0000 7fa38ccc2700 1 -- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/4233713404 conn(0x7fa36c033690 msgr2=0x7fa36c035a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:15.289 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.285+0000 7fa38ccc2700 1 --2- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/4233713404 conn(0x7fa36c033690 0x7fa36c035a80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fa37c003720 tx=0x7fa37c006d10).stop 2021-03-09T08:56:15.290 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.285+0000 7fa38ccc2700 1 -- 172.21.15.5:0/3629699819 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3880f9b20 msgr2=0x7fa388108270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:15.290 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.285+0000 7fa38ccc2700 1 --2- 172.21.15.5:0/3629699819 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3880f9b20 0x7fa388108270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fa378009420 tx=0x7fa378006ee0).stop 2021-03-09T08:56:15.290 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.285+0000 7fa38ccc2700 1 -- 172.21.15.5:0/3629699819 shutdown_connections 2021-03-09T08:56:15.291 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.285+0000 7fa38ccc2700 1 --2- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/4233713404 conn(0x7fa36c033690 0x7fa36c035a80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:15.291 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.285+0000 7fa38ccc2700 1 --2- 172.21.15.5:0/3629699819 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3880f9b20 0x7fa388108270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:15.291 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.285+0000 7fa38ccc2700 1 --2- 172.21.15.5:0/3629699819 >> v2:172.21.15.5:6800/167667630 conn(0x7fa36c02fe20 0x7fa36c0322e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:15.291 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.285+0000 7fa38ccc2700 1 -- 172.21.15.5:0/3629699819 >> 172.21.15.5:0/3629699819 conn(0x7fa3880f5300 msgr2=0x7fa3880f6170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:15.292 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.285+0000 7fa38ccc2700 1 -- 172.21.15.5:0/3629699819 shutdown_connections 2021-03-09T08:56:15.292 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:15.285+0000 7fa38ccc2700 1 -- 172.21.15.5:0/3629699819 wait complete. 2021-03-09T08:56:15.543 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: cluster 2021-03-09T08:56:14.213741+0000 mon.a (mon.0) 30 : cluster [INF] Active manager daemon y restarted 2021-03-09T08:56:15.543 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: cluster 2021-03-09T08:56:14.214444+0000 mon.a (mon.0) 31 : cluster [INF] Activating manager daemon y 2021-03-09T08:56:15.543 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: cluster 2021-03-09T08:56:14.217920+0000 mon.a (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2021-03-09T08:56:15.544 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: cluster 2021-03-09T08:56:14.270526+0000 mon.a (mon.0) 33 : cluster [DBG] mgrmap e6: y(active, starting, since 0.0561826s) 2021-03-09T08:56:15.544 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: audit 2021-03-09T08:56:14.273264+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-03-09T08:56:15.544 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: audit 2021-03-09T08:56:14.275205+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-03-09T08:56:15.544 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: audit 2021-03-09T08:56:14.276696+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-03-09T08:56:15.545 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: audit 2021-03-09T08:56:14.277179+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-03-09T08:56:15.545 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: audit 2021-03-09T08:56:14.277628+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-03-09T08:56:15.545 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: cluster 2021-03-09T08:56:14.282372+0000 mon.a (mon.0) 39 : cluster [INF] Manager daemon y is now available 2021-03-09T08:56:15.545 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: audit 2021-03-09T08:56:14.307515+0000 mon.a (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' 2021-03-09T08:56:15.545 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: audit 2021-03-09T08:56:14.311879+0000 mon.a (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' 2021-03-09T08:56:15.546 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: audit 2021-03-09T08:56:14.317532+0000 mon.a (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' 2021-03-09T08:56:15.546 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: audit 2021-03-09T08:56:14.452120+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:56:15.546 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: audit 2021-03-09T08:56:14.456893+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:56:15.546 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: audit 2021-03-09T08:56:14.480687+0000 mon.a (mon.0) 45 : audit [INF] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-09T08:56:15.547 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:15 smithi005 bash[12738]: audit 2021-03-09T08:56:14.483197+0000 mon.a (mon.0) 46 : audit [INF] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-09T08:56:16.042 INFO:teuthology.orchestra.run.smithi005.stderr:mgr epoch 5 is available 2021-03-09T08:56:16.042 INFO:teuthology.orchestra.run.smithi005.stderr:Setting orchestrator backend to cephadm... 2021-03-09T08:56:16.042 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 orch set backend cephadm 2021-03-09T08:56:16.363 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:16 smithi005 bash[12738]: cluster 2021-03-09T08:56:15.273136+0000 mon.a (mon.0) 47 : cluster [DBG] mgrmap e7: y(active, since 1.05879s) 2021-03-09T08:56:16.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:16 smithi005 bash[12738]: audit 2021-03-09T08:56:15.273579+0000 mgr.y (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-03-09T08:56:16.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:16 smithi005 bash[12738]: audit 2021-03-09T08:56:15.290112+0000 mgr.y (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-03-09T08:56:17.216 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.213+0000 7ff94f7b2700 1 Processor -- start 2021-03-09T08:56:17.216 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.213+0000 7ff94f7b2700 1 -- start start 2021-03-09T08:56:17.216 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.213+0000 7ff94f7b2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480fbd90 0x7ff9480fc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:17.217 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.213+0000 7ff94f7b2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9480fc6a0 con 0x7ff9480fbd90 2021-03-09T08:56:17.217 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff94d54e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480fbd90 0x7ff9480fc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:17.217 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff94d54e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480fbd90 0x7ff9480fc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46348/0 (socket says 172.21.15.5:46348) 2021-03-09T08:56:17.217 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff94d54e700 1 -- 172.21.15.5:0/1137391706 learned_addr learned my addr 172.21.15.5:0/1137391706 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:17.219 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff94d54e700 1 -- 172.21.15.5:0/1137391706 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9480fc7e0 con 0x7ff9480fbd90 2021-03-09T08:56:17.219 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff94d54e700 1 --2- 172.21.15.5:0/1137391706 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480fbd90 0x7ff9480fc1b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7ff93c00b950 tx=0x7ff93c006ee0).ready entity=mon.0 client_cookie=92140e72fd3367a0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:17.219 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff947fff700 1 -- 172.21.15.5:0/1137391706 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff93c009840 con 0x7ff9480fbd90 2021-03-09T08:56:17.220 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff947fff700 1 -- 172.21.15.5:0/1137391706 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7ff93c009e00 con 0x7ff9480fbd90 2021-03-09T08:56:17.220 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff947fff700 1 -- 172.21.15.5:0/1137391706 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff93c011c80 con 0x7ff9480fbd90 2021-03-09T08:56:17.220 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff94f7b2700 1 -- 172.21.15.5:0/1137391706 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480fbd90 msgr2=0x7ff9480fc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:17.220 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff94f7b2700 1 --2- 172.21.15.5:0/1137391706 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480fbd90 0x7ff9480fc1b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7ff93c00b950 tx=0x7ff93c006ee0).stop 2021-03-09T08:56:17.220 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff94f7b2700 1 -- 172.21.15.5:0/1137391706 shutdown_connections 2021-03-09T08:56:17.221 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff94f7b2700 1 --2- 172.21.15.5:0/1137391706 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480fbd90 0x7ff9480fc1b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:17.222 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff94f7b2700 1 -- 172.21.15.5:0/1137391706 >> 172.21.15.5:0/1137391706 conn(0x7ff94806f450 msgr2=0x7ff94806f860 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:17.223 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff94f7b2700 1 -- 172.21.15.5:0/1137391706 shutdown_connections 2021-03-09T08:56:17.223 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff94f7b2700 1 -- 172.21.15.5:0/1137391706 wait complete. 2021-03-09T08:56:17.223 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff94f7b2700 1 Processor -- start 2021-03-09T08:56:17.223 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff94f7b2700 1 -- start start 2021-03-09T08:56:17.224 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff94f7b2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480fbd90 0x7ff94810a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:17.224 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.217+0000 7ff94f7b2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94810a8a0 con 0x7ff9480fbd90 2021-03-09T08:56:17.224 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.221+0000 7ff94d54e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480fbd90 0x7ff94810a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:17.225 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.221+0000 7ff94d54e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480fbd90 0x7ff94810a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46350/0 (socket says 172.21.15.5:46350) 2021-03-09T08:56:17.226 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.221+0000 7ff94d54e700 1 -- 172.21.15.5:0/3282143021 learned_addr learned my addr 172.21.15.5:0/3282143021 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:17.226 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.221+0000 7ff94d54e700 1 -- 172.21.15.5:0/3282143021 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94810a9e0 con 0x7ff9480fbd90 2021-03-09T08:56:17.226 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.221+0000 7ff94d54e700 1 --2- 172.21.15.5:0/3282143021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480fbd90 0x7ff94810a3b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff93c009fd0 tx=0x7ff93c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:17.226 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.221+0000 7ff945ffb700 1 -- 172.21.15.5:0/3282143021 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff93c009ca0 con 0x7ff9480fbd90 2021-03-09T08:56:17.227 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.221+0000 7ff945ffb700 1 -- 172.21.15.5:0/3282143021 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7ff93c011400 con 0x7ff9480fbd90 2021-03-09T08:56:17.227 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.221+0000 7ff94f7b2700 1 -- 172.21.15.5:0/3282143021 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94810ac30 con 0x7ff9480fbd90 2021-03-09T08:56:17.227 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.221+0000 7ff945ffb700 1 -- 172.21.15.5:0/3282143021 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff93c01bc00 con 0x7ff9480fbd90 2021-03-09T08:56:17.227 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.221+0000 7ff94f7b2700 1 -- 172.21.15.5:0/3282143021 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94810b790 con 0x7ff9480fbd90 2021-03-09T08:56:17.228 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.221+0000 7ff94f7b2700 1 -- 172.21.15.5:0/3282143021 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9480589f0 con 0x7ff9480fbd90 2021-03-09T08:56:17.229 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.225+0000 7ff945ffb700 1 -- 172.21.15.5:0/3282143021 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 8) v1 ==== 36060+0+0 (secure 0 0 0) 0x7ff93c02b070 con 0x7ff9480fbd90 2021-03-09T08:56:17.229 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.225+0000 7ff945ffb700 1 --2- 172.21.15.5:0/3282143021 >> v2:172.21.15.5:6800/4233713404 conn(0x7ff93402fe20 0x7ff9340322e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:17.229 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.225+0000 7ff945ffb700 1 -- 172.21.15.5:0/3282143021 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7ff93c041c20 con 0x7ff9480fbd90 2021-03-09T08:56:17.229 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.225+0000 7ff94cd4d700 1 --2- 172.21.15.5:0/3282143021 >> v2:172.21.15.5:6800/4233713404 conn(0x7ff93402fe20 0x7ff9340322e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:17.230 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.225+0000 7ff94cd4d700 1 --2- 172.21.15.5:0/3282143021 >> v2:172.21.15.5:6800/4233713404 conn(0x7ff93402fe20 0x7ff9340322e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff9380089e0 tx=0x7ff938006ec0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:17.230 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.229+0000 7ff945ffb700 1 -- 172.21.15.5:0/3282143021 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7ff93c01b620 con 0x7ff9480fbd90 2021-03-09T08:56:17.589 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.589+0000 7ff94f7b2700 1 -- 172.21.15.5:0/3282143021 --> v2:172.21.15.5:6800/4233713404 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9480f9f10 con 0x7ff93402fe20 2021-03-09T08:56:17.601 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.601+0000 7ff945ffb700 1 -- 172.21.15.5:0/3282143021 <== mgr.14116 v2:172.21.15.5:6800/4233713404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff9480f9f10 con 0x7ff93402fe20 2021-03-09T08:56:17.604 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.601+0000 7ff94f7b2700 1 -- 172.21.15.5:0/3282143021 >> v2:172.21.15.5:6800/4233713404 conn(0x7ff93402fe20 msgr2=0x7ff9340322e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:17.604 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.605+0000 7ff94f7b2700 1 --2- 172.21.15.5:0/3282143021 >> v2:172.21.15.5:6800/4233713404 conn(0x7ff93402fe20 0x7ff9340322e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff9380089e0 tx=0x7ff938006ec0).stop 2021-03-09T08:56:17.606 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.605+0000 7ff94f7b2700 1 -- 172.21.15.5:0/3282143021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480fbd90 msgr2=0x7ff94810a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:17.606 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.605+0000 7ff94f7b2700 1 --2- 172.21.15.5:0/3282143021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480fbd90 0x7ff94810a3b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff93c009fd0 tx=0x7ff93c006ee0).stop 2021-03-09T08:56:17.606 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.605+0000 7ff94f7b2700 1 -- 172.21.15.5:0/3282143021 shutdown_connections 2021-03-09T08:56:17.607 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.605+0000 7ff94f7b2700 1 --2- 172.21.15.5:0/3282143021 >> v2:172.21.15.5:6800/4233713404 conn(0x7ff93402fe20 0x7ff9340322e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:17.607 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.605+0000 7ff94f7b2700 1 --2- 172.21.15.5:0/3282143021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9480fbd90 0x7ff94810a3b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:17.607 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.605+0000 7ff94f7b2700 1 -- 172.21.15.5:0/3282143021 >> 172.21.15.5:0/3282143021 conn(0x7ff94806f450 msgr2=0x7ff9480f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:17.607 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.605+0000 7ff94f7b2700 1 -- 172.21.15.5:0/3282143021 shutdown_connections 2021-03-09T08:56:17.608 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:17.605+0000 7ff94f7b2700 1 -- 172.21.15.5:0/3282143021 wait complete. 2021-03-09T08:56:17.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:17 smithi005 bash[12738]: cluster 2021-03-09T08:56:16.275073+0000 mon.a (mon.0) 48 : cluster [DBG] mgrmap e8: y(active, since 2s) 2021-03-09T08:56:18.331 INFO:teuthology.orchestra.run.smithi005.stderr:Generating ssh key... 2021-03-09T08:56:18.332 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 cephadm generate-key 2021-03-09T08:56:18.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:18 smithi005 bash[12738]: audit 2021-03-09T08:56:17.594753+0000 mgr.y (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:56:18.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:18 smithi005 bash[12738]: audit 2021-03-09T08:56:17.604378+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' 2021-03-09T08:56:18.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:18 smithi005 bash[12738]: audit 2021-03-09T08:56:17.610270+0000 mon.a (mon.0) 50 : audit [DBG] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:56:19.557 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.557+0000 7f80d2c9a700 1 Processor -- start 2021-03-09T08:56:19.558 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.557+0000 7f80d2c9a700 1 -- start start 2021-03-09T08:56:19.559 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.557+0000 7f80d2c9a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80cc0faf60 0x7f80cc0fb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:19.559 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.557+0000 7f80d2c9a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc0fb870 con 0x7f80cc0faf60 2021-03-09T08:56:19.559 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.557+0000 7f80d1c98700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80cc0faf60 0x7f80cc0fb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:19.559 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.557+0000 7f80d1c98700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80cc0faf60 0x7f80cc0fb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46354/0 (socket says 172.21.15.5:46354) 2021-03-09T08:56:19.560 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.557+0000 7f80d1c98700 1 -- 172.21.15.5:0/3286034628 learned_addr learned my addr 172.21.15.5:0/3286034628 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:19.560 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.557+0000 7f80d1c98700 1 -- 172.21.15.5:0/3286034628 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80cc0fb9b0 con 0x7f80cc0faf60 2021-03-09T08:56:19.560 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.557+0000 7f80d1c98700 1 --2- 172.21.15.5:0/3286034628 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80cc0faf60 0x7f80cc0fb380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f80c000b950 tx=0x7f80c0006ee0).ready entity=mon.0 client_cookie=a284e348fabac1ff server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:19.560 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.557+0000 7f80d0c96700 1 -- 172.21.15.5:0/3286034628 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80c0009840 con 0x7f80cc0faf60 2021-03-09T08:56:19.562 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.557+0000 7f80d0c96700 1 -- 172.21.15.5:0/3286034628 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7f80c0009e00 con 0x7f80cc0faf60 2021-03-09T08:56:19.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.557+0000 7f80d0c96700 1 -- 172.21.15.5:0/3286034628 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80c0011c80 con 0x7f80cc0faf60 2021-03-09T08:56:19.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.561+0000 7f80d2c9a700 1 -- 172.21.15.5:0/3286034628 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80cc0faf60 msgr2=0x7f80cc0fb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:19.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.561+0000 7f80d2c9a700 1 --2- 172.21.15.5:0/3286034628 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80cc0faf60 0x7f80cc0fb380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f80c000b950 tx=0x7f80c0006ee0).stop 2021-03-09T08:56:19.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.561+0000 7f80d2c9a700 1 -- 172.21.15.5:0/3286034628 shutdown_connections 2021-03-09T08:56:19.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.561+0000 7f80d2c9a700 1 --2- 172.21.15.5:0/3286034628 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80cc0faf60 0x7f80cc0fb380 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:19.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.561+0000 7f80d2c9a700 1 -- 172.21.15.5:0/3286034628 >> 172.21.15.5:0/3286034628 conn(0x7f80cc0f37a0 msgr2=0x7f80cc0f5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:19.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.561+0000 7f80d2c9a700 1 -- 172.21.15.5:0/3286034628 shutdown_connections 2021-03-09T08:56:19.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.561+0000 7f80d2c9a700 1 -- 172.21.15.5:0/3286034628 wait complete. 2021-03-09T08:56:19.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.561+0000 7f80d2c9a700 1 Processor -- start 2021-03-09T08:56:19.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.561+0000 7f80d2c9a700 1 -- start start 2021-03-09T08:56:19.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.561+0000 7f80d2c9a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80cc0faf60 0x7f80cc107fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:19.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.561+0000 7f80d2c9a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc1084d0 con 0x7f80cc0faf60 2021-03-09T08:56:19.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.561+0000 7f80d1c98700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80cc0faf60 0x7f80cc107fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:19.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.561+0000 7f80d1c98700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80cc0faf60 0x7f80cc107fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46356/0 (socket says 172.21.15.5:46356) 2021-03-09T08:56:19.567 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.561+0000 7f80d1c98700 1 -- 172.21.15.5:0/1217967242 learned_addr learned my addr 172.21.15.5:0/1217967242 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:19.568 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.561+0000 7f80d1c98700 1 -- 172.21.15.5:0/1217967242 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80cc108610 con 0x7f80cc0faf60 2021-03-09T08:56:19.568 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.561+0000 7f80d1c98700 1 --2- 172.21.15.5:0/1217967242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80cc0faf60 0x7f80cc107fe0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f80c0009fd0 tx=0x7f80c0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:19.568 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.565+0000 7f80ca7fc700 1 -- 172.21.15.5:0/1217967242 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80c0009ca0 con 0x7f80cc0faf60 2021-03-09T08:56:19.568 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.565+0000 7f80ca7fc700 1 -- 172.21.15.5:0/1217967242 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7f80c0011400 con 0x7f80cc0faf60 2021-03-09T08:56:19.569 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.565+0000 7f80d2c9a700 1 -- 172.21.15.5:0/1217967242 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80cc108860 con 0x7f80cc0faf60 2021-03-09T08:56:19.569 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.565+0000 7f80ca7fc700 1 -- 172.21.15.5:0/1217967242 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80c001bc00 con 0x7f80cc0faf60 2021-03-09T08:56:19.569 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.565+0000 7f80d2c9a700 1 -- 172.21.15.5:0/1217967242 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80cc1093c0 con 0x7f80cc0faf60 2021-03-09T08:56:19.569 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.565+0000 7f80ca7fc700 1 -- 172.21.15.5:0/1217967242 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 8) v1 ==== 36060+0+0 (secure 0 0 0) 0x7f80c000f070 con 0x7f80cc0faf60 2021-03-09T08:56:19.569 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.565+0000 7f80ca7fc700 1 --2- 172.21.15.5:0/1217967242 >> v2:172.21.15.5:6800/4233713404 conn(0x7f80b802fe20 0x7f80b80322e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:19.570 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.565+0000 7f80ca7fc700 1 -- 172.21.15.5:0/1217967242 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f80c0041840 con 0x7f80cc0faf60 2021-03-09T08:56:19.570 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.565+0000 7f80d1497700 1 --2- 172.21.15.5:0/1217967242 >> v2:172.21.15.5:6800/4233713404 conn(0x7f80b802fe20 0x7f80b80322e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:19.570 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.565+0000 7f80d2c9a700 1 -- 172.21.15.5:0/1217967242 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80cc0589f0 con 0x7f80cc0faf60 2021-03-09T08:56:19.570 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.565+0000 7f80d1497700 1 --2- 172.21.15.5:0/1217967242 >> v2:172.21.15.5:6800/4233713404 conn(0x7f80b802fe20 0x7f80b80322e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f80bc0089e0 tx=0x7f80bc006ec0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:19.578 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.577+0000 7f80ca7fc700 1 -- 172.21.15.5:0/1217967242 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f80c0016070 con 0x7f80cc0faf60 2021-03-09T08:56:19.878 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:19.877+0000 7f80d2c9a700 1 -- 172.21.15.5:0/1217967242 --> v2:172.21.15.5:6800/4233713404 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f80cc0008d0 con 0x7f80b802fe20 2021-03-09T08:56:20.080 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:20.077+0000 7f80ca7fc700 1 -- 172.21.15.5:0/1217967242 <== mgr.14116 v2:172.21.15.5:6800/4233713404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f80cc0008d0 con 0x7f80b802fe20 2021-03-09T08:56:20.083 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:20.081+0000 7f80d2c9a700 1 -- 172.21.15.5:0/1217967242 >> v2:172.21.15.5:6800/4233713404 conn(0x7f80b802fe20 msgr2=0x7f80b80322e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:20.083 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:20.081+0000 7f80d2c9a700 1 --2- 172.21.15.5:0/1217967242 >> v2:172.21.15.5:6800/4233713404 conn(0x7f80b802fe20 0x7f80b80322e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f80bc0089e0 tx=0x7f80bc006ec0).stop 2021-03-09T08:56:20.084 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:20.081+0000 7f80d2c9a700 1 -- 172.21.15.5:0/1217967242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80cc0faf60 msgr2=0x7f80cc107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:20.084 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:20.081+0000 7f80d2c9a700 1 --2- 172.21.15.5:0/1217967242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80cc0faf60 0x7f80cc107fe0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f80c0009fd0 tx=0x7f80c0006ee0).stop 2021-03-09T08:56:20.084 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:20.081+0000 7f80d2c9a700 1 -- 172.21.15.5:0/1217967242 shutdown_connections 2021-03-09T08:56:20.084 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:20.081+0000 7f80d2c9a700 1 --2- 172.21.15.5:0/1217967242 >> v2:172.21.15.5:6800/4233713404 conn(0x7f80b802fe20 0x7f80b80322e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:20.085 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:20.081+0000 7f80d2c9a700 1 --2- 172.21.15.5:0/1217967242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80cc0faf60 0x7f80cc107fe0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:20.085 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:20.081+0000 7f80d2c9a700 1 -- 172.21.15.5:0/1217967242 >> 172.21.15.5:0/1217967242 conn(0x7f80cc0f37a0 msgr2=0x7f80cc0fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:20.085 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:20.081+0000 7f80d2c9a700 1 -- 172.21.15.5:0/1217967242 shutdown_connections 2021-03-09T08:56:20.085 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:20.081+0000 7f80d2c9a700 1 -- 172.21.15.5:0/1217967242 wait complete. 2021-03-09T08:56:20.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12738]: cephadm 2021-03-09T08:56:19.882899+0000 mgr.y (mgr.14116) 5 : cephadm [INF] Generating ssh key... 2021-03-09T08:56:20.308 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: Generating public/private rsa key pair. 2021-03-09T08:56:20.308 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: Your identification has been saved in /tmp/tmpp0oxpnn_/key. 2021-03-09T08:56:20.308 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: Your public key has been saved in /tmp/tmpp0oxpnn_/key.pub. 2021-03-09T08:56:20.309 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: The key fingerprint is: 2021-03-09T08:56:20.309 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: SHA256:xwo43nceXyDTxYMCyYSNlvGAd5U4EUyvo1dZq+DUHwE ceph-252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:56:20.309 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: The key's randomart image is: 2021-03-09T08:56:20.309 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: +---[RSA 3072]----+ 2021-03-09T08:56:20.309 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: | .o@**E. | 2021-03-09T08:56:20.310 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: | . *oOo.. o | 2021-03-09T08:56:20.310 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: | o ...o + + | 2021-03-09T08:56:20.310 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: | . + = + . | 2021-03-09T08:56:20.310 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: | o . S O = | 2021-03-09T08:56:20.311 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: | . o = * = o | 2021-03-09T08:56:20.311 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: | . o = + . . | 2021-03-09T08:56:20.311 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: | o o o . | 2021-03-09T08:56:20.311 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: | . . | 2021-03-09T08:56:20.311 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:20 smithi005 bash[12977]: +----[SHA256]-----+ 2021-03-09T08:56:20.780 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 cephadm get-pub-key 2021-03-09T08:56:21.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:21 smithi005 bash[12738]: audit 2021-03-09T08:56:20.077478+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' 2021-03-09T08:56:21.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:21 smithi005 bash[12738]: audit 2021-03-09T08:56:20.082031+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' 2021-03-09T08:56:21.907 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.905+0000 7f433292d700 1 Processor -- start 2021-03-09T08:56:21.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.905+0000 7f433292d700 1 -- start start 2021-03-09T08:56:21.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.905+0000 7f433292d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f432c0f9940 0x7f432c0f9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:21.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.905+0000 7f433292d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c0fa250 con 0x7f432c0f9940 2021-03-09T08:56:21.908 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.905+0000 7f433192b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f432c0f9940 0x7f432c0f9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:21.909 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.905+0000 7f433192b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f432c0f9940 0x7f432c0f9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46360/0 (socket says 172.21.15.5:46360) 2021-03-09T08:56:21.909 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.905+0000 7f433192b700 1 -- 172.21.15.5:0/3460865339 learned_addr learned my addr 172.21.15.5:0/3460865339 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:21.909 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.905+0000 7f433192b700 1 -- 172.21.15.5:0/3460865339 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c0fa390 con 0x7f432c0f9940 2021-03-09T08:56:21.910 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.905+0000 7f433192b700 1 --2- 172.21.15.5:0/3460865339 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f432c0f9940 0x7f432c0f9d60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f431c009cc0 tx=0x7f431c0092d0).ready entity=mon.0 client_cookie=210fdad82b926fa6 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:21.914 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.905+0000 7f4330929700 1 -- 172.21.15.5:0/3460865339 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f431c00bbf0 con 0x7f432c0f9940 2021-03-09T08:56:21.914 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.905+0000 7f4330929700 1 -- 172.21.15.5:0/3460865339 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7f431c00bd50 con 0x7f432c0f9940 2021-03-09T08:56:21.914 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.905+0000 7f4330929700 1 -- 172.21.15.5:0/3460865339 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f431c010560 con 0x7f432c0f9940 2021-03-09T08:56:21.915 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.909+0000 7f433292d700 1 -- 172.21.15.5:0/3460865339 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f432c0f9940 msgr2=0x7f432c0f9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:21.915 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.909+0000 7f433292d700 1 --2- 172.21.15.5:0/3460865339 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f432c0f9940 0x7f432c0f9d60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f431c009cc0 tx=0x7f431c0092d0).stop 2021-03-09T08:56:21.915 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.909+0000 7f433292d700 1 -- 172.21.15.5:0/3460865339 shutdown_connections 2021-03-09T08:56:21.916 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.909+0000 7f433292d700 1 --2- 172.21.15.5:0/3460865339 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f432c0f9940 0x7f432c0f9d60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:21.916 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.909+0000 7f433292d700 1 -- 172.21.15.5:0/3460865339 >> 172.21.15.5:0/3460865339 conn(0x7f432c0f51a0 msgr2=0x7f432c0f75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:21.916 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.909+0000 7f433292d700 1 -- 172.21.15.5:0/3460865339 shutdown_connections 2021-03-09T08:56:21.916 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.909+0000 7f433292d700 1 -- 172.21.15.5:0/3460865339 wait complete. 2021-03-09T08:56:21.917 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.909+0000 7f433292d700 1 Processor -- start 2021-03-09T08:56:21.917 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.909+0000 7f433292d700 1 -- start start 2021-03-09T08:56:21.917 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.909+0000 7f433292d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f432c0f9940 0x7f432c108070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:21.918 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.909+0000 7f433292d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c108560 con 0x7f432c0f9940 2021-03-09T08:56:21.918 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.909+0000 7f433192b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f432c0f9940 0x7f432c108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:21.918 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.909+0000 7f433192b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f432c0f9940 0x7f432c108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46362/0 (socket says 172.21.15.5:46362) 2021-03-09T08:56:21.918 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.909+0000 7f433192b700 1 -- 172.21.15.5:0/2608415379 learned_addr learned my addr 172.21.15.5:0/2608415379 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:21.919 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.913+0000 7f433192b700 1 -- 172.21.15.5:0/2608415379 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c1086a0 con 0x7f432c0f9940 2021-03-09T08:56:21.919 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.913+0000 7f433192b700 1 --2- 172.21.15.5:0/2608415379 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f432c0f9940 0x7f432c108070 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f431c006b80 tx=0x7f431c0092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:21.919 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.913+0000 7f432a7fc700 1 -- 172.21.15.5:0/2608415379 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f431c00bbf0 con 0x7f432c0f9940 2021-03-09T08:56:21.919 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.913+0000 7f432a7fc700 1 -- 172.21.15.5:0/2608415379 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7f431c00bec0 con 0x7f432c0f9940 2021-03-09T08:56:21.920 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.913+0000 7f433292d700 1 -- 172.21.15.5:0/2608415379 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432c1088f0 con 0x7f432c0f9940 2021-03-09T08:56:21.922 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.913+0000 7f432a7fc700 1 -- 172.21.15.5:0/2608415379 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f431c023490 con 0x7f432c0f9940 2021-03-09T08:56:21.922 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.913+0000 7f433292d700 1 -- 172.21.15.5:0/2608415379 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432c109450 con 0x7f432c0f9940 2021-03-09T08:56:21.922 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.913+0000 7f432a7fc700 1 -- 172.21.15.5:0/2608415379 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 8) v1 ==== 36060+0+0 (secure 0 0 0) 0x7f431c00f070 con 0x7f432c0f9940 2021-03-09T08:56:21.923 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.913+0000 7f432a7fc700 1 --2- 172.21.15.5:0/2608415379 >> v2:172.21.15.5:6800/4233713404 conn(0x7f431802fe20 0x7f43180322e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:21.923 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.913+0000 7f432a7fc700 1 -- 172.21.15.5:0/2608415379 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f431c041e30 con 0x7f432c0f9940 2021-03-09T08:56:21.923 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.913+0000 7f433112a700 1 --2- 172.21.15.5:0/2608415379 >> v2:172.21.15.5:6800/4233713404 conn(0x7f431802fe20 0x7f43180322e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:21.924 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.913+0000 7f433292d700 1 -- 172.21.15.5:0/2608415379 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f432c0589f0 con 0x7f432c0f9940 2021-03-09T08:56:21.924 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.917+0000 7f433112a700 1 --2- 172.21.15.5:0/2608415379 >> v2:172.21.15.5:6800/4233713404 conn(0x7f431802fe20 0x7f43180322e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f43200095f0 tx=0x7f4320006b30).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:21.925 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:21.921+0000 7f432a7fc700 1 -- 172.21.15.5:0/2608415379 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f431c01e020 con 0x7f432c0f9940 2021-03-09T08:56:22.222 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:22.221+0000 7f433292d700 1 -- 172.21.15.5:0/2608415379 --> v2:172.21.15.5:6800/4233713404 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f432c0f6700 con 0x7f431802fe20 2021-03-09T08:56:22.223 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:22.221+0000 7f432a7fc700 1 -- 172.21.15.5:0/2608415379 <== mgr.14116 v2:172.21.15.5:6800/4233713404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f432c0f6700 con 0x7f431802fe20 2021-03-09T08:56:22.223 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQ59cFYEB+n2raXUNxQMtzJ1oRz3Z8Xit3FDgyZfGXT4n3oRJRGrsp7HH9szPwTTjOZMmPe2WmK8mGa+KAxbIF+Knl6tgTFZrvqeSS7ZZ0nrhVwwi42jmoxPZ7qoK2dECiztxJ/CpkIR82ZLl8VktDT7pRtIhnNgQ6yoTWYH+7a55M2IDsCzcPWwwHoSpteo+g3o4NIJyRX4IZBNuTGqM3995dpR6TAr40FGL5zYNEPXUPcE9mXquVxvSeEFgDMsWLoVMKamEV4WoYdUTZFzFrXzPKN4Ddz+/GliHWFkIIkx9GG4Nf6eISQssiHF+1hJEUt5ao6rIL5dWi4NfC+NmVbWXspGP/KfQJS71Y1sFHE9C2WVI6fs4Drv07Y7qb+0EspKq3o3sFt/uBf3FCIFfExGEWk3qDtT/FGqOkpwA85m81ws0e6aUG8rG1+Lfvo0ETlP+BpQekr6T0SVleo1Qvq9iaLXnuu6rWxmclg/Yftm7mjELtNzTmrABW3hxFzk0= ceph-252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:56:22.226 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:22.225+0000 7f433292d700 1 -- 172.21.15.5:0/2608415379 >> v2:172.21.15.5:6800/4233713404 conn(0x7f431802fe20 msgr2=0x7f43180322e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:22.226 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:22.225+0000 7f433292d700 1 --2- 172.21.15.5:0/2608415379 >> v2:172.21.15.5:6800/4233713404 conn(0x7f431802fe20 0x7f43180322e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f43200095f0 tx=0x7f4320006b30).stop 2021-03-09T08:56:22.227 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:22.225+0000 7f433292d700 1 -- 172.21.15.5:0/2608415379 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f432c0f9940 msgr2=0x7f432c108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:22.227 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:22.225+0000 7f433292d700 1 --2- 172.21.15.5:0/2608415379 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f432c0f9940 0x7f432c108070 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f431c006b80 tx=0x7f431c0092d0).stop 2021-03-09T08:56:22.227 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:22.225+0000 7f433292d700 1 -- 172.21.15.5:0/2608415379 shutdown_connections 2021-03-09T08:56:22.227 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:22.225+0000 7f433292d700 1 --2- 172.21.15.5:0/2608415379 >> v2:172.21.15.5:6800/4233713404 conn(0x7f431802fe20 0x7f43180322e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:22.227 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:22.225+0000 7f433292d700 1 --2- 172.21.15.5:0/2608415379 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f432c0f9940 0x7f432c108070 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:22.228 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:22.225+0000 7f433292d700 1 -- 172.21.15.5:0/2608415379 >> 172.21.15.5:0/2608415379 conn(0x7f432c0f51a0 msgr2=0x7f432c0f5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:22.228 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:22.225+0000 7f433292d700 1 -- 172.21.15.5:0/2608415379 shutdown_connections 2021-03-09T08:56:22.228 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:22.225+0000 7f433292d700 1 -- 172.21.15.5:0/2608415379 wait complete. 2021-03-09T08:56:22.886 INFO:teuthology.orchestra.run.smithi005.stderr:Wrote public SSH key to to /home/ubuntu/cephtest/ceph.pub 2021-03-09T08:56:22.887 INFO:teuthology.orchestra.run.smithi005.stderr:Adding key to root@localhost authorized_keys... 2021-03-09T08:56:22.888 INFO:teuthology.orchestra.run.smithi005.stderr:Adding host smithi005... 2021-03-09T08:56:22.888 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 orch host add smithi005 2021-03-09T08:56:23.363 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:23 smithi005 bash[12738]: audit 2021-03-09T08:56:22.226302+0000 mgr.y (mgr.14116) 6 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:56:24.020 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.017+0000 7fb1e07b0700 1 Processor -- start 2021-03-09T08:56:24.021 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1e07b0700 1 -- start start 2021-03-09T08:56:24.022 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1e07b0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d806fcb0 0x7fb1d806e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:24.022 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1e07b0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d80700d0 con 0x7fb1d806fcb0 2021-03-09T08:56:24.022 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1de54c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d806fcb0 0x7fb1d806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:24.023 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1de54c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d806fcb0 0x7fb1d806e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46366/0 (socket says 172.21.15.5:46366) 2021-03-09T08:56:24.023 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1de54c700 1 -- 172.21.15.5:0/1583113405 learned_addr learned my addr 172.21.15.5:0/1583113405 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:24.025 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1de54c700 1 -- 172.21.15.5:0/1583113405 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d806e740 con 0x7fb1d806fcb0 2021-03-09T08:56:24.025 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1de54c700 1 --2- 172.21.15.5:0/1583113405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d806fcb0 0x7fb1d806e250 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb1c800b950 tx=0x7fb1c8006ee0).ready entity=mon.0 client_cookie=c8c104b14d354c93 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:24.025 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1dd54a700 1 -- 172.21.15.5:0/1583113405 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1c8009840 con 0x7fb1d806fcb0 2021-03-09T08:56:24.025 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1dd54a700 1 -- 172.21.15.5:0/1583113405 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7fb1c8009e00 con 0x7fb1d806fcb0 2021-03-09T08:56:24.025 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1dd54a700 1 -- 172.21.15.5:0/1583113405 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1c8011c80 con 0x7fb1d806fcb0 2021-03-09T08:56:24.026 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1e07b0700 1 -- 172.21.15.5:0/1583113405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d806fcb0 msgr2=0x7fb1d806e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:24.026 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1e07b0700 1 --2- 172.21.15.5:0/1583113405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d806fcb0 0x7fb1d806e250 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb1c800b950 tx=0x7fb1c8006ee0).stop 2021-03-09T08:56:24.026 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1e07b0700 1 -- 172.21.15.5:0/1583113405 shutdown_connections 2021-03-09T08:56:24.026 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1e07b0700 1 --2- 172.21.15.5:0/1583113405 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d806fcb0 0x7fb1d806e250 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:24.027 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1e07b0700 1 -- 172.21.15.5:0/1583113405 >> 172.21.15.5:0/1583113405 conn(0x7fb1d80f5300 msgr2=0x7fb1d80f7720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:24.027 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1e07b0700 1 -- 172.21.15.5:0/1583113405 shutdown_connections 2021-03-09T08:56:24.027 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1e07b0700 1 -- 172.21.15.5:0/1583113405 wait complete. 2021-03-09T08:56:24.030 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.021+0000 7fb1e07b0700 1 Processor -- start 2021-03-09T08:56:24.030 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1e07b0700 1 -- start start 2021-03-09T08:56:24.030 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1e07b0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d806fcb0 0x7fb1d80f9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:24.030 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1e07b0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d80f9fa0 con 0x7fb1d806fcb0 2021-03-09T08:56:24.031 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1de54c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d806fcb0 0x7fb1d80f9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:24.031 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1de54c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d806fcb0 0x7fb1d80f9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46368/0 (socket says 172.21.15.5:46368) 2021-03-09T08:56:24.031 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1de54c700 1 -- 172.21.15.5:0/262468242 learned_addr learned my addr 172.21.15.5:0/262468242 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:24.031 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1de54c700 1 -- 172.21.15.5:0/262468242 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d80fa0e0 con 0x7fb1d806fcb0 2021-03-09T08:56:24.031 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1de54c700 1 --2- 172.21.15.5:0/262468242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d806fcb0 0x7fb1d80f9ab0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fb1c8009fd0 tx=0x7fb1c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:24.032 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1c6ffd700 1 -- 172.21.15.5:0/262468242 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1c8009ca0 con 0x7fb1d806fcb0 2021-03-09T08:56:24.032 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1c6ffd700 1 -- 172.21.15.5:0/262468242 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7fb1c8011ae0 con 0x7fb1d806fcb0 2021-03-09T08:56:24.032 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1e07b0700 1 -- 172.21.15.5:0/262468242 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d80fa330 con 0x7fb1d806fcb0 2021-03-09T08:56:24.032 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1c6ffd700 1 -- 172.21.15.5:0/262468242 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1c801b9b0 con 0x7fb1d806fcb0 2021-03-09T08:56:24.032 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1e07b0700 1 -- 172.21.15.5:0/262468242 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d80faee0 con 0x7fb1d806fcb0 2021-03-09T08:56:24.033 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1c6ffd700 1 -- 172.21.15.5:0/262468242 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 8) v1 ==== 36060+0+0 (secure 0 0 0) 0x7fb1c800f070 con 0x7fb1d806fcb0 2021-03-09T08:56:24.033 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1c6ffd700 1 --2- 172.21.15.5:0/262468242 >> v2:172.21.15.5:6800/4233713404 conn(0x7fb1c002fe20 0x7fb1c00322e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:24.033 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1c6ffd700 1 -- 172.21.15.5:0/262468242 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7fb1c80413d0 con 0x7fb1d806fcb0 2021-03-09T08:56:24.033 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.025+0000 7fb1ddd4b700 1 --2- 172.21.15.5:0/262468242 >> v2:172.21.15.5:6800/4233713404 conn(0x7fb1c002fe20 0x7fb1c00322e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:24.033 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.029+0000 7fb1e07b0700 1 -- 172.21.15.5:0/262468242 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d80589f0 con 0x7fb1d806fcb0 2021-03-09T08:56:24.034 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.033+0000 7fb1ddd4b700 1 --2- 172.21.15.5:0/262468242 >> v2:172.21.15.5:6800/4233713404 conn(0x7fb1c002fe20 0x7fb1c00322e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb1d40095f0 tx=0x7fb1d4006b30).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:24.035 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.033+0000 7fb1c6ffd700 1 -- 172.21.15.5:0/262468242 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7fb1c8016030 con 0x7fb1d806fcb0 2021-03-09T08:56:24.367 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:24.365+0000 7fb1e07b0700 1 -- 172.21.15.5:0/262468242 --> v2:172.21.15.5:6800/4233713404 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi005", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1d80f7630 con 0x7fb1c002fe20 2021-03-09T08:56:24.863 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:24 smithi005 bash[12977]: Warning: Permanently added 'smithi005,172.21.15.5' (ECDSA) to the list of known hosts. 2021-03-09T08:56:25.155 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:25.153+0000 7fb1c6ffd700 1 -- 172.21.15.5:0/262468242 <== mgr.14116 v2:172.21.15.5:6800/4233713404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fb1d80f7630 con 0x7fb1c002fe20 2021-03-09T08:56:25.156 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: Added host 'smithi005' 2021-03-09T08:56:25.159 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:25.157+0000 7fb1e07b0700 1 -- 172.21.15.5:0/262468242 >> v2:172.21.15.5:6800/4233713404 conn(0x7fb1c002fe20 msgr2=0x7fb1c00322e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:25.159 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:25.157+0000 7fb1e07b0700 1 --2- 172.21.15.5:0/262468242 >> v2:172.21.15.5:6800/4233713404 conn(0x7fb1c002fe20 0x7fb1c00322e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb1d40095f0 tx=0x7fb1d4006b30).stop 2021-03-09T08:56:25.159 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:25.157+0000 7fb1e07b0700 1 -- 172.21.15.5:0/262468242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d806fcb0 msgr2=0x7fb1d80f9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:25.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:25.157+0000 7fb1e07b0700 1 --2- 172.21.15.5:0/262468242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d806fcb0 0x7fb1d80f9ab0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fb1c8009fd0 tx=0x7fb1c8006ee0).stop 2021-03-09T08:56:25.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:25.157+0000 7fb1e07b0700 1 -- 172.21.15.5:0/262468242 shutdown_connections 2021-03-09T08:56:25.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:25.157+0000 7fb1e07b0700 1 --2- 172.21.15.5:0/262468242 >> v2:172.21.15.5:6800/4233713404 conn(0x7fb1c002fe20 0x7fb1c00322e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:25.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:25.157+0000 7fb1e07b0700 1 --2- 172.21.15.5:0/262468242 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d806fcb0 0x7fb1d80f9ab0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:25.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:25.157+0000 7fb1e07b0700 1 -- 172.21.15.5:0/262468242 >> 172.21.15.5:0/262468242 conn(0x7fb1d80f5300 msgr2=0x7fb1d80f69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:25.161 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:25.157+0000 7fb1e07b0700 1 -- 172.21.15.5:0/262468242 shutdown_connections 2021-03-09T08:56:25.161 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:25.157+0000 7fb1e07b0700 1 -- 172.21.15.5:0/262468242 wait complete. 2021-03-09T08:56:25.349 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:25 smithi005 bash[12738]: audit 2021-03-09T08:56:24.372774+0000 mgr.y (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi005", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:56:25.995 INFO:teuthology.orchestra.run.smithi005.stderr:Deploying unmanaged mon service... 2021-03-09T08:56:25.996 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 orch apply mon --unmanaged 2021-03-09T08:56:26.363 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:26 smithi005 bash[12738]: audit 2021-03-09T08:56:25.158216+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' 2021-03-09T08:56:26.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:26 smithi005 bash[12738]: cephadm 2021-03-09T08:56:25.158933+0000 mgr.y (mgr.14116) 8 : cephadm [INF] Added host smithi005 2021-03-09T08:56:26.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:26 smithi005 bash[12738]: audit 2021-03-09T08:56:25.159411+0000 mon.a (mon.0) 54 : audit [DBG] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:56:27.781 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92cffff700 1 Processor -- start 2021-03-09T08:56:27.781 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92cffff700 1 -- start start 2021-03-09T08:56:27.781 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92cffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92d00fbb40 0x7f92d00fbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:27.782 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92cffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d00fc450 con 0x7f92d00fbb40 2021-03-09T08:56:27.783 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92ceffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92d00fbb40 0x7f92d00fbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:27.783 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92ceffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92d00fbb40 0x7f92d00fbf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46380/0 (socket says 172.21.15.5:46380) 2021-03-09T08:56:27.783 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92ceffd700 1 -- 172.21.15.5:0/2972324818 learned_addr learned my addr 172.21.15.5:0/2972324818 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:27.783 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92ceffd700 1 -- 172.21.15.5:0/2972324818 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d00fc590 con 0x7f92d00fbb40 2021-03-09T08:56:27.787 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92ceffd700 1 --2- 172.21.15.5:0/2972324818 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92d00fbb40 0x7f92d00fbf60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f92c000b950 tx=0x7f92c0006ee0).ready entity=mon.0 client_cookie=faa8a697c203c330 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:27.787 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92cdffb700 1 -- 172.21.15.5:0/2972324818 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92c0009840 con 0x7f92d00fbb40 2021-03-09T08:56:27.787 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92cdffb700 1 -- 172.21.15.5:0/2972324818 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7f92c0009e00 con 0x7f92d00fbb40 2021-03-09T08:56:27.787 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92cdffb700 1 -- 172.21.15.5:0/2972324818 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92c0011c80 con 0x7f92d00fbb40 2021-03-09T08:56:27.788 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92cffff700 1 -- 172.21.15.5:0/2972324818 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92d00fbb40 msgr2=0x7f92d00fbf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:27.788 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92cffff700 1 --2- 172.21.15.5:0/2972324818 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92d00fbb40 0x7f92d00fbf60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f92c000b950 tx=0x7f92c0006ee0).stop 2021-03-09T08:56:27.788 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92cffff700 1 -- 172.21.15.5:0/2972324818 shutdown_connections 2021-03-09T08:56:27.788 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92cffff700 1 --2- 172.21.15.5:0/2972324818 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92d00fbb40 0x7f92d00fbf60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:27.788 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92cffff700 1 -- 172.21.15.5:0/2972324818 >> 172.21.15.5:0/2972324818 conn(0x7f92d006f2f0 msgr2=0x7f92d006f700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:27.789 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92cffff700 1 -- 172.21.15.5:0/2972324818 shutdown_connections 2021-03-09T08:56:27.789 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.781+0000 7f92cffff700 1 -- 172.21.15.5:0/2972324818 wait complete. 2021-03-09T08:56:27.789 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.785+0000 7f92cffff700 1 Processor -- start 2021-03-09T08:56:27.789 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.785+0000 7f92cffff700 1 -- start start 2021-03-09T08:56:27.790 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.785+0000 7f92cffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92d00fbb40 0x7f92d010a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:27.790 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.785+0000 7f92cffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d010a700 con 0x7f92d00fbb40 2021-03-09T08:56:27.790 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.785+0000 7f92ceffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92d00fbb40 0x7f92d010a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:27.790 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.785+0000 7f92ceffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92d00fbb40 0x7f92d010a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46382/0 (socket says 172.21.15.5:46382) 2021-03-09T08:56:27.791 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.785+0000 7f92ceffd700 1 -- 172.21.15.5:0/3663434515 learned_addr learned my addr 172.21.15.5:0/3663434515 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:27.791 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.785+0000 7f92ceffd700 1 -- 172.21.15.5:0/3663434515 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d010a840 con 0x7f92d00fbb40 2021-03-09T08:56:27.791 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.785+0000 7f92ceffd700 1 --2- 172.21.15.5:0/3663434515 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92d00fbb40 0x7f92d010a210 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f92c0009fd0 tx=0x7f92c0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:27.791 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.785+0000 7f92c77fe700 1 -- 172.21.15.5:0/3663434515 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92c0009ca0 con 0x7f92d00fbb40 2021-03-09T08:56:27.791 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.785+0000 7f92c77fe700 1 -- 172.21.15.5:0/3663434515 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1001+0+0 (secure 0 0 0) 0x7f92c0011400 con 0x7f92d00fbb40 2021-03-09T08:56:27.792 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.785+0000 7f92cffff700 1 -- 172.21.15.5:0/3663434515 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d010aa90 con 0x7f92d00fbb40 2021-03-09T08:56:27.792 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.785+0000 7f92c77fe700 1 -- 172.21.15.5:0/3663434515 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92c001bc00 con 0x7f92d00fbb40 2021-03-09T08:56:27.792 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.785+0000 7f92cffff700 1 -- 172.21.15.5:0/3663434515 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d010b5f0 con 0x7f92d00fbb40 2021-03-09T08:56:27.793 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.789+0000 7f92c77fe700 1 -- 172.21.15.5:0/3663434515 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 8) v1 ==== 36060+0+0 (secure 0 0 0) 0x7f92c002b070 con 0x7f92d00fbb40 2021-03-09T08:56:27.793 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.789+0000 7f92c77fe700 1 --2- 172.21.15.5:0/3663434515 >> v2:172.21.15.5:6800/4233713404 conn(0x7f92b002fe70 0x7f92b0032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:27.794 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.789+0000 7f92cffff700 1 -- 172.21.15.5:0/3663434515 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d00589f0 con 0x7f92d00fbb40 2021-03-09T08:56:27.794 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.789+0000 7f92ce7fc700 1 --2- 172.21.15.5:0/3663434515 >> v2:172.21.15.5:6800/4233713404 conn(0x7f92b002fe70 0x7f92b0032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:27.794 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.789+0000 7f92c77fe700 1 -- 172.21.15.5:0/3663434515 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f92c0042540 con 0x7f92d00fbb40 2021-03-09T08:56:27.794 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.789+0000 7f92ce7fc700 1 --2- 172.21.15.5:0/3663434515 >> v2:172.21.15.5:6800/4233713404 conn(0x7f92b002fe70 0x7f92b0032330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f92b80096a0 tx=0x7f92b8006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:27.795 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:27.793+0000 7f92c77fe700 1 -- 172.21.15.5:0/3663434515 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f92c0022d70 con 0x7f92d00fbb40 2021-03-09T08:56:28.145 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:28.145+0000 7f92cffff700 1 -- 172.21.15.5:0/3663434515 --> v2:172.21.15.5:6800/4233713404 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f92d00face0 con 0x7f92b002fe70 2021-03-09T08:56:28.153 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:28.153+0000 7f92c77fe700 1 -- 172.21.15.5:0/3663434515 <== mgr.14116 v2:172.21.15.5:6800/4233713404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f92d00face0 con 0x7f92b002fe70 2021-03-09T08:56:28.153 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: Scheduled mon update... 2021-03-09T08:56:28.155 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:28.153+0000 7f92cffff700 1 -- 172.21.15.5:0/3663434515 >> v2:172.21.15.5:6800/4233713404 conn(0x7f92b002fe70 msgr2=0x7f92b0032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:28.155 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:28.153+0000 7f92cffff700 1 --2- 172.21.15.5:0/3663434515 >> v2:172.21.15.5:6800/4233713404 conn(0x7f92b002fe70 0x7f92b0032330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f92b80096a0 tx=0x7f92b8006b40).stop 2021-03-09T08:56:28.157 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:28.153+0000 7f92cffff700 1 -- 172.21.15.5:0/3663434515 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92d00fbb40 msgr2=0x7f92d010a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:28.157 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:28.153+0000 7f92cffff700 1 --2- 172.21.15.5:0/3663434515 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92d00fbb40 0x7f92d010a210 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f92c0009fd0 tx=0x7f92c0006ee0).stop 2021-03-09T08:56:28.157 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:28.153+0000 7f92cffff700 1 -- 172.21.15.5:0/3663434515 shutdown_connections 2021-03-09T08:56:28.157 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:28.153+0000 7f92cffff700 1 --2- 172.21.15.5:0/3663434515 >> v2:172.21.15.5:6800/4233713404 conn(0x7f92b002fe70 0x7f92b0032330 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:28.157 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:28.153+0000 7f92cffff700 1 --2- 172.21.15.5:0/3663434515 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92d00fbb40 0x7f92d010a210 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:28.158 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:28.153+0000 7f92cffff700 1 -- 172.21.15.5:0/3663434515 >> 172.21.15.5:0/3663434515 conn(0x7f92d006f2f0 msgr2=0x7f92d00fa050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:28.158 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:28.157+0000 7f92cffff700 1 -- 172.21.15.5:0/3663434515 shutdown_connections 2021-03-09T08:56:28.158 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:28.157+0000 7f92cffff700 1 -- 172.21.15.5:0/3663434515 wait complete. 2021-03-09T08:56:29.193 INFO:teuthology.orchestra.run.smithi005.stderr:Deploying unmanaged mgr service... 2021-03-09T08:56:29.193 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 orch apply mgr --unmanaged 2021-03-09T08:56:29.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:29 smithi005 bash[12738]: audit 2021-03-09T08:56:28.150736+0000 mgr.y (mgr.14116) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:56:29.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:29 smithi005 bash[12738]: cephadm 2021-03-09T08:56:28.151740+0000 mgr.y (mgr.14116) 10 : cephadm [INF] Saving service mon spec with placement count:5 2021-03-09T08:56:29.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:29 smithi005 bash[12738]: audit 2021-03-09T08:56:28.156323+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' 2021-03-09T08:56:29.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:29 smithi005 bash[12738]: audit 2021-03-09T08:56:28.999668+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' 2021-03-09T08:56:30.416 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.412+0000 7f2e1f59e700 1 Processor -- start 2021-03-09T08:56:30.416 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.412+0000 7f2e1f59e700 1 -- start start 2021-03-09T08:56:30.417 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.412+0000 7f2e1f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e200f9940 0x7f2e200f9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:30.417 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.412+0000 7f2e1f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e200fa250 con 0x7f2e200f9940 2021-03-09T08:56:30.417 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.412+0000 7f2e1e59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e200f9940 0x7f2e200f9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:30.417 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.412+0000 7f2e1e59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e200f9940 0x7f2e200f9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46386/0 (socket says 172.21.15.5:46386) 2021-03-09T08:56:30.418 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.412+0000 7f2e1e59c700 1 -- 172.21.15.5:0/982923249 learned_addr learned my addr 172.21.15.5:0/982923249 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:30.419 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1e59c700 1 -- 172.21.15.5:0/982923249 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e200fa390 con 0x7f2e200f9940 2021-03-09T08:56:30.420 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1e59c700 1 --2- 172.21.15.5:0/982923249 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e200f9940 0x7f2e200f9d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f2e0800b950 tx=0x7f2e08006ee0).ready entity=mon.0 client_cookie=e527b3ba8e76209a server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:30.420 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1d59a700 1 -- 172.21.15.5:0/982923249 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e08009840 con 0x7f2e200f9940 2021-03-09T08:56:30.420 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1d59a700 1 -- 172.21.15.5:0/982923249 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f2e08009e20 con 0x7f2e200f9940 2021-03-09T08:56:30.420 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1d59a700 1 -- 172.21.15.5:0/982923249 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e08011d10 con 0x7f2e200f9940 2021-03-09T08:56:30.421 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1f59e700 1 -- 172.21.15.5:0/982923249 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e200f9940 msgr2=0x7f2e200f9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:30.421 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1f59e700 1 --2- 172.21.15.5:0/982923249 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e200f9940 0x7f2e200f9d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f2e0800b950 tx=0x7f2e08006ee0).stop 2021-03-09T08:56:30.421 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1f59e700 1 -- 172.21.15.5:0/982923249 shutdown_connections 2021-03-09T08:56:30.421 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1f59e700 1 --2- 172.21.15.5:0/982923249 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e200f9940 0x7f2e200f9d60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:30.421 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1f59e700 1 -- 172.21.15.5:0/982923249 >> 172.21.15.5:0/982923249 conn(0x7f2e200f51a0 msgr2=0x7f2e200f75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:30.422 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1f59e700 1 -- 172.21.15.5:0/982923249 shutdown_connections 2021-03-09T08:56:30.422 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1f59e700 1 -- 172.21.15.5:0/982923249 wait complete. 2021-03-09T08:56:30.422 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1f59e700 1 Processor -- start 2021-03-09T08:56:30.424 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1f59e700 1 -- start start 2021-03-09T08:56:30.424 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e2006fd50 0x7f2e2006e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:30.424 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e20070170 con 0x7f2e2006fd50 2021-03-09T08:56:30.425 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1e59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e2006fd50 0x7f2e2006e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:30.425 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1e59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e2006fd50 0x7f2e2006e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46388/0 (socket says 172.21.15.5:46388) 2021-03-09T08:56:30.425 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.416+0000 7f2e1e59c700 1 -- 172.21.15.5:0/1974727262 learned_addr learned my addr 172.21.15.5:0/1974727262 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:30.425 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.420+0000 7f2e1e59c700 1 -- 172.21.15.5:0/1974727262 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2006e700 con 0x7f2e2006fd50 2021-03-09T08:56:30.425 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.420+0000 7f2e1e59c700 1 --2- 172.21.15.5:0/1974727262 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e2006fd50 0x7f2e2006e210 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f2e08013000 tx=0x7f2e08006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:30.426 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.420+0000 7f2e16ffd700 1 -- 172.21.15.5:0/1974727262 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e08009ca0 con 0x7f2e2006fd50 2021-03-09T08:56:30.426 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.420+0000 7f2e16ffd700 1 -- 172.21.15.5:0/1974727262 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f2e08011b50 con 0x7f2e2006fd50 2021-03-09T08:56:30.426 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.420+0000 7f2e16ffd700 1 -- 172.21.15.5:0/1974727262 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e0801bb20 con 0x7f2e2006fd50 2021-03-09T08:56:30.426 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.420+0000 7f2e1f59e700 1 -- 172.21.15.5:0/1974727262 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2006e950 con 0x7f2e2006fd50 2021-03-09T08:56:30.427 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.420+0000 7f2e1f59e700 1 -- 172.21.15.5:0/1974727262 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2006ede0 con 0x7f2e2006fd50 2021-03-09T08:56:30.428 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.420+0000 7f2e1f59e700 1 -- 172.21.15.5:0/1974727262 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e200589f0 con 0x7f2e2006fd50 2021-03-09T08:56:30.428 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.424+0000 7f2e16ffd700 1 -- 172.21.15.5:0/1974727262 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 8) v1 ==== 36060+0+0 (secure 0 0 0) 0x7f2e0800f070 con 0x7f2e2006fd50 2021-03-09T08:56:30.428 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.424+0000 7f2e16ffd700 1 --2- 172.21.15.5:0/1974727262 >> v2:172.21.15.5:6800/4233713404 conn(0x7f2e0002fe20 0x7f2e000322e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:30.429 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.424+0000 7f2e16ffd700 1 -- 172.21.15.5:0/1974727262 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f2e08040f30 con 0x7f2e2006fd50 2021-03-09T08:56:30.429 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.424+0000 7f2e1dd9b700 1 --2- 172.21.15.5:0/1974727262 >> v2:172.21.15.5:6800/4233713404 conn(0x7f2e0002fe20 0x7f2e000322e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:30.429 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.424+0000 7f2e1dd9b700 1 --2- 172.21.15.5:0/1974727262 >> v2:172.21.15.5:6800/4233713404 conn(0x7f2e0002fe20 0x7f2e000322e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f2e10008a70 tx=0x7f2e10006eb0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:30.432 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.428+0000 7f2e16ffd700 1 -- 172.21.15.5:0/1974727262 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f2e08019300 con 0x7f2e2006fd50 2021-03-09T08:56:30.781 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.780+0000 7f2e1f59e700 1 -- 172.21.15.5:0/1974727262 --> v2:172.21.15.5:6800/4233713404 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f2e200275d0 con 0x7f2e0002fe20 2021-03-09T08:56:30.789 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.784+0000 7f2e16ffd700 1 -- 172.21.15.5:0/1974727262 <== mgr.14116 v2:172.21.15.5:6800/4233713404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2e200275d0 con 0x7f2e0002fe20 2021-03-09T08:56:30.789 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: Scheduled mgr update... 2021-03-09T08:56:30.791 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.788+0000 7f2e1f59e700 1 -- 172.21.15.5:0/1974727262 >> v2:172.21.15.5:6800/4233713404 conn(0x7f2e0002fe20 msgr2=0x7f2e000322e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:30.791 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.788+0000 7f2e1f59e700 1 --2- 172.21.15.5:0/1974727262 >> v2:172.21.15.5:6800/4233713404 conn(0x7f2e0002fe20 0x7f2e000322e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f2e10008a70 tx=0x7f2e10006eb0).stop 2021-03-09T08:56:30.792 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.788+0000 7f2e1f59e700 1 -- 172.21.15.5:0/1974727262 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e2006fd50 msgr2=0x7f2e2006e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:30.792 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.788+0000 7f2e1f59e700 1 --2- 172.21.15.5:0/1974727262 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e2006fd50 0x7f2e2006e210 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f2e08013000 tx=0x7f2e08006ee0).stop 2021-03-09T08:56:30.792 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.788+0000 7f2e1f59e700 1 -- 172.21.15.5:0/1974727262 shutdown_connections 2021-03-09T08:56:30.792 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.788+0000 7f2e1f59e700 1 --2- 172.21.15.5:0/1974727262 >> v2:172.21.15.5:6800/4233713404 conn(0x7f2e0002fe20 0x7f2e000322e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:30.793 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.788+0000 7f2e1f59e700 1 --2- 172.21.15.5:0/1974727262 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2e2006fd50 0x7f2e2006e210 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:30.793 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.788+0000 7f2e1f59e700 1 -- 172.21.15.5:0/1974727262 >> 172.21.15.5:0/1974727262 conn(0x7f2e200f51a0 msgr2=0x7f2e200f6950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:30.793 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.788+0000 7f2e1f59e700 1 -- 172.21.15.5:0/1974727262 shutdown_connections 2021-03-09T08:56:30.793 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:30.788+0000 7f2e1f59e700 1 -- 172.21.15.5:0/1974727262 wait complete. 2021-03-09T08:56:30.851 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:30 smithi005 bash[12738]: audit 2021-03-09T08:56:29.440554+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' 2021-03-09T08:56:31.517 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 config set mgr mgr/cephadm/container_init True --force 2021-03-09T08:56:31.865 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:31 smithi005 bash[12738]: audit 2021-03-09T08:56:30.786096+0000 mgr.y (mgr.14116) 11 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:56:31.865 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:31 smithi005 bash[12738]: cephadm 2021-03-09T08:56:30.787323+0000 mgr.y (mgr.14116) 12 : cephadm [INF] Saving service mgr spec with placement count:2 2021-03-09T08:56:31.865 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:31 smithi005 bash[12738]: audit 2021-03-09T08:56:30.790504+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' 2021-03-09T08:56:32.671 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.671+0000 7f8a0e37e700 1 Processor -- start 2021-03-09T08:56:32.671 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.671+0000 7f8a0e37e700 1 -- start start 2021-03-09T08:56:32.673 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.671+0000 7f8a0e37e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a0806dc50 0x7f8a080fc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:32.673 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.671+0000 7f8a0e37e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a080fca70 con 0x7f8a0806dc50 2021-03-09T08:56:32.673 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.671+0000 7f8a0d37c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a0806dc50 0x7f8a080fc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:32.674 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.671+0000 7f8a0d37c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a0806dc50 0x7f8a080fc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46392/0 (socket says 172.21.15.5:46392) 2021-03-09T08:56:32.674 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.671+0000 7f8a0d37c700 1 -- 172.21.15.5:0/1050052237 learned_addr learned my addr 172.21.15.5:0/1050052237 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:32.674 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.671+0000 7f8a0d37c700 1 -- 172.21.15.5:0/1050052237 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a080fcbb0 con 0x7f8a0806dc50 2021-03-09T08:56:32.674 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.671+0000 7f8a0d37c700 1 --2- 172.21.15.5:0/1050052237 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a0806dc50 0x7f8a080fc580 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f8a0400b950 tx=0x7f8a04006ee0).ready entity=mon.0 client_cookie=f40c1edfbf515938 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:32.675 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.671+0000 7f89fbfff700 1 -- 172.21.15.5:0/1050052237 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a04009840 con 0x7f8a0806dc50 2021-03-09T08:56:32.675 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.671+0000 7f89fbfff700 1 -- 172.21.15.5:0/1050052237 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8a04009e20 con 0x7f8a0806dc50 2021-03-09T08:56:32.675 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.671+0000 7f89fbfff700 1 -- 172.21.15.5:0/1050052237 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a04011d10 con 0x7f8a0806dc50 2021-03-09T08:56:32.677 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.671+0000 7f8a0e37e700 1 -- 172.21.15.5:0/1050052237 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a0806dc50 msgr2=0x7f8a080fc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:32.677 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.671+0000 7f8a0e37e700 1 --2- 172.21.15.5:0/1050052237 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a0806dc50 0x7f8a080fc580 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f8a0400b950 tx=0x7f8a04006ee0).stop 2021-03-09T08:56:32.677 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f8a0e37e700 1 -- 172.21.15.5:0/1050052237 shutdown_connections 2021-03-09T08:56:32.677 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f8a0e37e700 1 --2- 172.21.15.5:0/1050052237 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a0806dc50 0x7f8a080fc580 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:32.678 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f8a0e37e700 1 -- 172.21.15.5:0/1050052237 >> 172.21.15.5:0/1050052237 conn(0x7f8a080f4020 msgr2=0x7f8a080f6440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:32.678 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f8a0e37e700 1 -- 172.21.15.5:0/1050052237 shutdown_connections 2021-03-09T08:56:32.678 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f8a0e37e700 1 -- 172.21.15.5:0/1050052237 wait complete. 2021-03-09T08:56:32.679 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f8a0e37e700 1 Processor -- start 2021-03-09T08:56:32.680 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f8a0e37e700 1 -- start start 2021-03-09T08:56:32.681 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f8a0e37e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a0806dc50 0x7f8a0810a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:32.681 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f8a0e37e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0810a760 con 0x7f8a0806dc50 2021-03-09T08:56:32.681 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f8a0d37c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a0806dc50 0x7f8a0810a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:32.682 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f8a0d37c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a0806dc50 0x7f8a0810a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46394/0 (socket says 172.21.15.5:46394) 2021-03-09T08:56:32.682 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f8a0d37c700 1 -- 172.21.15.5:0/2962290296 learned_addr learned my addr 172.21.15.5:0/2962290296 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:32.682 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f8a0d37c700 1 -- 172.21.15.5:0/2962290296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0810a8a0 con 0x7f8a0806dc50 2021-03-09T08:56:32.682 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f8a0d37c700 1 --2- 172.21.15.5:0/2962290296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a0806dc50 0x7f8a0810a270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f8a04009420 tx=0x7f8a04006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:32.683 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f89fa7fc700 1 -- 172.21.15.5:0/2962290296 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a04009ca0 con 0x7f8a0806dc50 2021-03-09T08:56:32.683 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f89fa7fc700 1 -- 172.21.15.5:0/2962290296 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8a04010f00 con 0x7f8a0806dc50 2021-03-09T08:56:32.683 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f8a0e37e700 1 -- 172.21.15.5:0/2962290296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a0810aaf0 con 0x7f8a0806dc50 2021-03-09T08:56:32.683 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f89fa7fc700 1 -- 172.21.15.5:0/2962290296 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a0401bc80 con 0x7f8a0806dc50 2021-03-09T08:56:32.684 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.675+0000 7f8a0e37e700 1 -- 172.21.15.5:0/2962290296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a0810b650 con 0x7f8a0806dc50 2021-03-09T08:56:32.684 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.679+0000 7f89fa7fc700 1 -- 172.21.15.5:0/2962290296 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 8) v1 ==== 36060+0+0 (secure 0 0 0) 0x7f8a0400f070 con 0x7f8a0806dc50 2021-03-09T08:56:32.684 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.679+0000 7f8a0e37e700 1 -- 172.21.15.5:0/2962290296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a080589f0 con 0x7f8a0806dc50 2021-03-09T08:56:32.684 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.679+0000 7f89fa7fc700 1 --2- 172.21.15.5:0/2962290296 >> v2:172.21.15.5:6800/4233713404 conn(0x7f89ec02fde0 0x7f89ec0322a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:32.684 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.679+0000 7f89fa7fc700 1 -- 172.21.15.5:0/2962290296 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f8a04041420 con 0x7f8a0806dc50 2021-03-09T08:56:32.685 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.679+0000 7f8a0cb7b700 1 --2- 172.21.15.5:0/2962290296 >> v2:172.21.15.5:6800/4233713404 conn(0x7f89ec02fde0 0x7f89ec0322a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:32.685 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.679+0000 7f8a0cb7b700 1 --2- 172.21.15.5:0/2962290296 >> v2:172.21.15.5:6800/4233713404 conn(0x7f89ec02fde0 0x7f89ec0322a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f89fc0096a0 tx=0x7f89fc006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:32.687 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.687+0000 7f89fa7fc700 1 -- 172.21.15.5:0/2962290296 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f8a04019d60 con 0x7f8a0806dc50 2021-03-09T08:56:32.987 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.986+0000 7f8a0e37e700 1 -- 172.21.15.5:0/2962290296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f8a0810b950 con 0x7f8a0806dc50 2021-03-09T08:56:32.994 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.990+0000 7f89fa7fc700 1 -- 172.21.15.5:0/2962290296 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f8a04016070 con 0x7f8a0806dc50 2021-03-09T08:56:32.996 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.994+0000 7f8a0e37e700 1 -- 172.21.15.5:0/2962290296 >> v2:172.21.15.5:6800/4233713404 conn(0x7f89ec02fde0 msgr2=0x7f89ec0322a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:32.997 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.994+0000 7f8a0e37e700 1 --2- 172.21.15.5:0/2962290296 >> v2:172.21.15.5:6800/4233713404 conn(0x7f89ec02fde0 0x7f89ec0322a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f89fc0096a0 tx=0x7f89fc006b40).stop 2021-03-09T08:56:32.997 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.994+0000 7f8a0e37e700 1 -- 172.21.15.5:0/2962290296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a0806dc50 msgr2=0x7f8a0810a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:32.997 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.994+0000 7f8a0e37e700 1 --2- 172.21.15.5:0/2962290296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a0806dc50 0x7f8a0810a270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f8a04009420 tx=0x7f8a04006ee0).stop 2021-03-09T08:56:32.998 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.994+0000 7f8a0e37e700 1 -- 172.21.15.5:0/2962290296 shutdown_connections 2021-03-09T08:56:32.999 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.994+0000 7f8a0e37e700 1 --2- 172.21.15.5:0/2962290296 >> v2:172.21.15.5:6800/4233713404 conn(0x7f89ec02fde0 0x7f89ec0322a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:32.999 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.994+0000 7f8a0e37e700 1 --2- 172.21.15.5:0/2962290296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a0806dc50 0x7f8a0810a270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:32.999 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.994+0000 7f8a0e37e700 1 -- 172.21.15.5:0/2962290296 >> 172.21.15.5:0/2962290296 conn(0x7f8a080f4020 msgr2=0x7f8a080f56b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:32.999 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.994+0000 7f8a0e37e700 1 -- 172.21.15.5:0/2962290296 shutdown_connections 2021-03-09T08:56:33.000 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:32.994+0000 7f8a0e37e700 1 -- 172.21.15.5:0/2962290296 wait complete. 2021-03-09T08:56:33.672 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 config set mgr mgr/dashboard/ssl_server_port 8443 2021-03-09T08:56:34.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:33 smithi005 bash[12738]: audit 2021-03-09T08:56:32.995343+0000 mon.a (mon.0) 59 : audit [INF] from='client.? 172.21.15.5:0/2962290296' entity='client.admin' 2021-03-09T08:56:34.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:33 smithi005 bash[12738]: audit 2021-03-09T08:56:33.793330+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.5:0/3855141192' entity='mgr.y' 2021-03-09T08:56:35.424 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11f57a5700 1 Processor -- start 2021-03-09T08:56:35.425 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11f57a5700 1 -- start start 2021-03-09T08:56:35.425 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11f57a5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11f00f7fc0 0x7f11f00f83e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:35.427 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11f57a5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f00f88d0 con 0x7f11f00f7fc0 2021-03-09T08:56:35.427 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11effff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11f00f7fc0 0x7f11f00f83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:35.427 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11effff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11f00f7fc0 0x7f11f00f83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46398/0 (socket says 172.21.15.5:46398) 2021-03-09T08:56:35.428 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11effff700 1 -- 172.21.15.5:0/3427246076 learned_addr learned my addr 172.21.15.5:0/3427246076 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:35.428 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11effff700 1 -- 172.21.15.5:0/3427246076 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f00f8a10 con 0x7f11f00f7fc0 2021-03-09T08:56:35.428 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11effff700 1 --2- 172.21.15.5:0/3427246076 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11f00f7fc0 0x7f11f00f83e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f11e000ca60 tx=0x7f11e0006ee0).ready entity=mon.0 client_cookie=fbd134fd589ade37 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:35.428 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11eeffd700 1 -- 172.21.15.5:0/3427246076 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f11e0009840 con 0x7f11f00f7fc0 2021-03-09T08:56:35.428 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11eeffd700 1 -- 172.21.15.5:0/3427246076 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f11e0009e20 con 0x7f11f00f7fc0 2021-03-09T08:56:35.429 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11eeffd700 1 -- 172.21.15.5:0/3427246076 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f11e0010dc0 con 0x7f11f00f7fc0 2021-03-09T08:56:35.429 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11f57a5700 1 -- 172.21.15.5:0/3427246076 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11f00f7fc0 msgr2=0x7f11f00f83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:35.429 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11f57a5700 1 --2- 172.21.15.5:0/3427246076 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11f00f7fc0 0x7f11f00f83e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f11e000ca60 tx=0x7f11e0006ee0).stop 2021-03-09T08:56:35.429 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11f57a5700 1 -- 172.21.15.5:0/3427246076 shutdown_connections 2021-03-09T08:56:35.430 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11f57a5700 1 --2- 172.21.15.5:0/3427246076 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11f00f7fc0 0x7f11f00f83e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:35.430 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11f57a5700 1 -- 172.21.15.5:0/3427246076 >> 172.21.15.5:0/3427246076 conn(0x7f11f00f37e0 msgr2=0x7f11f00f5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:35.430 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11f57a5700 1 -- 172.21.15.5:0/3427246076 shutdown_connections 2021-03-09T08:56:35.430 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.421+0000 7f11f57a5700 1 -- 172.21.15.5:0/3427246076 wait complete. 2021-03-09T08:56:35.433 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.425+0000 7f11f57a5700 1 Processor -- start 2021-03-09T08:56:35.433 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.425+0000 7f11f57a5700 1 -- start start 2021-03-09T08:56:35.433 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.425+0000 7f11f57a5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11f00f7fc0 0x7f11f0105ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:35.434 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.425+0000 7f11f57a5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f01063c0 con 0x7f11f00f7fc0 2021-03-09T08:56:35.434 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.425+0000 7f11effff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11f00f7fc0 0x7f11f0105ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:35.434 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.425+0000 7f11effff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11f00f7fc0 0x7f11f0105ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46400/0 (socket says 172.21.15.5:46400) 2021-03-09T08:56:35.434 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.425+0000 7f11effff700 1 -- 172.21.15.5:0/2546882945 learned_addr learned my addr 172.21.15.5:0/2546882945 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:35.434 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.425+0000 7f11effff700 1 -- 172.21.15.5:0/2546882945 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f0106500 con 0x7f11f00f7fc0 2021-03-09T08:56:35.435 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.425+0000 7f11effff700 1 --2- 172.21.15.5:0/2546882945 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11f00f7fc0 0x7f11f0105ed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f11e0006800 tx=0x7f11e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:35.435 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.425+0000 7f11ecff9700 1 -- 172.21.15.5:0/2546882945 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f11e000c1a0 con 0x7f11f00f7fc0 2021-03-09T08:56:35.435 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.425+0000 7f11f57a5700 1 -- 172.21.15.5:0/2546882945 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f0106750 con 0x7f11f00f7fc0 2021-03-09T08:56:35.435 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.425+0000 7f11ecff9700 1 -- 172.21.15.5:0/2546882945 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f11e001d070 con 0x7f11f00f7fc0 2021-03-09T08:56:35.436 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.425+0000 7f11ecff9700 1 -- 172.21.15.5:0/2546882945 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f11e0019e40 con 0x7f11f00f7fc0 2021-03-09T08:56:35.436 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.425+0000 7f11f57a5700 1 -- 172.21.15.5:0/2546882945 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f01071d0 con 0x7f11f00f7fc0 2021-03-09T08:56:35.436 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.425+0000 7f11ecff9700 1 -- 172.21.15.5:0/2546882945 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 8) v1 ==== 36060+0+0 (secure 0 0 0) 0x7f11e0029070 con 0x7f11f00f7fc0 2021-03-09T08:56:35.436 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.429+0000 7f11ecff9700 1 --2- 172.21.15.5:0/2546882945 >> v2:172.21.15.5:6800/4233713404 conn(0x7f11dc02fde0 0x7f11dc0322a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:35.436 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.429+0000 7f11ecff9700 1 -- 172.21.15.5:0/2546882945 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f11e0041860 con 0x7f11f00f7fc0 2021-03-09T08:56:35.437 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.429+0000 7f11ef7fe700 1 --2- 172.21.15.5:0/2546882945 >> v2:172.21.15.5:6800/4233713404 conn(0x7f11dc02fde0 0x7f11dc0322a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:35.437 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.429+0000 7f11f57a5700 1 -- 172.21.15.5:0/2546882945 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f00589f0 con 0x7f11f00f7fc0 2021-03-09T08:56:35.437 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.429+0000 7f11ef7fe700 1 --2- 172.21.15.5:0/2546882945 >> v2:172.21.15.5:6800/4233713404 conn(0x7f11dc02fde0 0x7f11dc0322a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f11e40096a0 tx=0x7f11e4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:35.438 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.433+0000 7f11ecff9700 1 -- 172.21.15.5:0/2546882945 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f11e0014080 con 0x7f11f00f7fc0 2021-03-09T08:56:35.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:35 smithi005 bash[12738]: cluster 2021-03-09T08:56:34.278408+0000 mgr.y (mgr.14116) 13 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:56:35.735 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.729+0000 7f11f57a5700 1 -- 172.21.15.5:0/2546882945 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f11f0045280 con 0x7f11f00f7fc0 2021-03-09T08:56:35.742 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.737+0000 7f11ecff9700 1 -- 172.21.15.5:0/2546882945 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7f11f0045280 con 0x7f11f00f7fc0 2021-03-09T08:56:35.746 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.741+0000 7f11f57a5700 1 -- 172.21.15.5:0/2546882945 >> v2:172.21.15.5:6800/4233713404 conn(0x7f11dc02fde0 msgr2=0x7f11dc0322a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:35.746 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.741+0000 7f11f57a5700 1 --2- 172.21.15.5:0/2546882945 >> v2:172.21.15.5:6800/4233713404 conn(0x7f11dc02fde0 0x7f11dc0322a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f11e40096a0 tx=0x7f11e4006b40).stop 2021-03-09T08:56:35.746 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.741+0000 7f11f57a5700 1 -- 172.21.15.5:0/2546882945 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11f00f7fc0 msgr2=0x7f11f0105ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:35.747 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.741+0000 7f11f57a5700 1 --2- 172.21.15.5:0/2546882945 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11f00f7fc0 0x7f11f0105ed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f11e0006800 tx=0x7f11e0006ee0).stop 2021-03-09T08:56:35.747 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.741+0000 7f11f57a5700 1 -- 172.21.15.5:0/2546882945 shutdown_connections 2021-03-09T08:56:35.747 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.741+0000 7f11f57a5700 1 --2- 172.21.15.5:0/2546882945 >> v2:172.21.15.5:6800/4233713404 conn(0x7f11dc02fde0 0x7f11dc0322a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:35.747 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.741+0000 7f11f57a5700 1 --2- 172.21.15.5:0/2546882945 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11f00f7fc0 0x7f11f0105ed0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:35.748 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.741+0000 7f11f57a5700 1 -- 172.21.15.5:0/2546882945 >> 172.21.15.5:0/2546882945 conn(0x7f11f00f37e0 msgr2=0x7f11f00fad40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:35.748 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.741+0000 7f11f57a5700 1 -- 172.21.15.5:0/2546882945 shutdown_connections 2021-03-09T08:56:35.748 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:35.741+0000 7f11f57a5700 1 -- 172.21.15.5:0/2546882945 wait complete. 2021-03-09T08:56:36.588 INFO:teuthology.orchestra.run.smithi005.stderr:Enabling the dashboard module... 2021-03-09T08:56:36.589 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 mgr module enable dashboard 2021-03-09T08:56:37.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:36 smithi005 bash[12738]: audit 2021-03-09T08:56:35.742349+0000 mon.a (mon.0) 61 : audit [INF] from='client.? 172.21.15.5:0/2546882945' entity='client.admin' 2021-03-09T08:56:37.778 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef80e5700 1 Processor -- start 2021-03-09T08:56:37.779 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef80e5700 1 -- start start 2021-03-09T08:56:37.779 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef80e5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef00f9030 0x7f8ef00f9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:37.779 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef80e5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef00f9940 con 0x7f8ef00f9030 2021-03-09T08:56:37.779 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef5e81700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef00f9030 0x7f8ef00f9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:37.780 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef5e81700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef00f9030 0x7f8ef00f9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46404/0 (socket says 172.21.15.5:46404) 2021-03-09T08:56:37.780 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef5e81700 1 -- 172.21.15.5:0/650281482 learned_addr learned my addr 172.21.15.5:0/650281482 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:37.782 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef5e81700 1 -- 172.21.15.5:0/650281482 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef00f9a80 con 0x7f8ef00f9030 2021-03-09T08:56:37.782 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef5e81700 1 --2- 172.21.15.5:0/650281482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef00f9030 0x7f8ef00f9450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f8ee0009cc0 tx=0x7f8ee00092d0).ready entity=mon.0 client_cookie=a9445fad6abc2b78 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:37.783 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef4e7f700 1 -- 172.21.15.5:0/650281482 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ee000bbf0 con 0x7f8ef00f9030 2021-03-09T08:56:37.783 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef4e7f700 1 -- 172.21.15.5:0/650281482 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8ee000bd50 con 0x7f8ef00f9030 2021-03-09T08:56:37.783 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef4e7f700 1 -- 172.21.15.5:0/650281482 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ee0010560 con 0x7f8ef00f9030 2021-03-09T08:56:37.783 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef80e5700 1 -- 172.21.15.5:0/650281482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef00f9030 msgr2=0x7f8ef00f9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:37.784 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef80e5700 1 --2- 172.21.15.5:0/650281482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef00f9030 0x7f8ef00f9450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f8ee0009cc0 tx=0x7f8ee00092d0).stop 2021-03-09T08:56:37.785 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef80e5700 1 -- 172.21.15.5:0/650281482 shutdown_connections 2021-03-09T08:56:37.785 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef80e5700 1 --2- 172.21.15.5:0/650281482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef00f9030 0x7f8ef00f9450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:37.785 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.772+0000 7f8ef80e5700 1 -- 172.21.15.5:0/650281482 >> 172.21.15.5:0/650281482 conn(0x7f8ef00f3960 msgr2=0x7f8ef00f5da0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:37.785 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.776+0000 7f8ef80e5700 1 -- 172.21.15.5:0/650281482 shutdown_connections 2021-03-09T08:56:37.786 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.776+0000 7f8ef80e5700 1 -- 172.21.15.5:0/650281482 wait complete. 2021-03-09T08:56:37.786 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.776+0000 7f8ef80e5700 1 Processor -- start 2021-03-09T08:56:37.787 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.776+0000 7f8ef80e5700 1 -- start start 2021-03-09T08:56:37.788 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.776+0000 7f8ef80e5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef00f9030 0x7f8ef0107170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:37.788 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.776+0000 7f8ef80e5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef0107660 con 0x7f8ef00f9030 2021-03-09T08:56:37.788 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.776+0000 7f8ef5e81700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef00f9030 0x7f8ef0107170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:37.788 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.776+0000 7f8ef5e81700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef00f9030 0x7f8ef0107170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46406/0 (socket says 172.21.15.5:46406) 2021-03-09T08:56:37.789 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.776+0000 7f8ef5e81700 1 -- 172.21.15.5:0/2460155537 learned_addr learned my addr 172.21.15.5:0/2460155537 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:37.789 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.776+0000 7f8ef5e81700 1 -- 172.21.15.5:0/2460155537 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef01077a0 con 0x7f8ef00f9030 2021-03-09T08:56:37.791 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.776+0000 7f8ef5e81700 1 --2- 172.21.15.5:0/2460155537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef00f9030 0x7f8ef0107170 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f8ee0006b80 tx=0x7f8ee00092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:37.791 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.776+0000 7f8eee7fc700 1 -- 172.21.15.5:0/2460155537 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ee00103a0 con 0x7f8ef00f9030 2021-03-09T08:56:37.791 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.776+0000 7f8eee7fc700 1 -- 172.21.15.5:0/2460155537 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8ee0011860 con 0x7f8ef00f9030 2021-03-09T08:56:37.791 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.776+0000 7f8ef80e5700 1 -- 172.21.15.5:0/2460155537 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef01079f0 con 0x7f8ef00f9030 2021-03-09T08:56:37.791 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.776+0000 7f8eee7fc700 1 -- 172.21.15.5:0/2460155537 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ee0022460 con 0x7f8ef00f9030 2021-03-09T08:56:37.792 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.776+0000 7f8ef80e5700 1 -- 172.21.15.5:0/2460155537 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef0108550 con 0x7f8ef00f9030 2021-03-09T08:56:37.792 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.780+0000 7f8eee7fc700 1 -- 172.21.15.5:0/2460155537 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 8) v1 ==== 36060+0+0 (secure 0 0 0) 0x7f8ee002a070 con 0x7f8ef00f9030 2021-03-09T08:56:37.792 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.780+0000 7f8eee7fc700 1 --2- 172.21.15.5:0/2460155537 >> v2:172.21.15.5:6800/4233713404 conn(0x7f8edc02fde0 0x7f8edc0322a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:37.792 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.780+0000 7f8eee7fc700 1 -- 172.21.15.5:0/2460155537 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f8ee0041e10 con 0x7f8ef00f9030 2021-03-09T08:56:37.793 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.780+0000 7f8ef80e5700 1 -- 172.21.15.5:0/2460155537 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef00589f0 con 0x7f8ef00f9030 2021-03-09T08:56:37.793 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.780+0000 7f8ef5680700 1 --2- 172.21.15.5:0/2460155537 >> v2:172.21.15.5:6800/4233713404 conn(0x7f8edc02fde0 0x7f8edc0322a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:37.793 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.780+0000 7f8ef5680700 1 --2- 172.21.15.5:0/2460155537 >> v2:172.21.15.5:6800/4233713404 conn(0x7f8edc02fde0 0x7f8edc0322a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f8ee40096a0 tx=0x7f8ee4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:37.793 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:37.784+0000 7f8eee7fc700 1 -- 172.21.15.5:0/2460155537 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f8ee001e080 con 0x7f8ef00f9030 2021-03-09T08:56:38.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:37 smithi005 bash[12738]: cluster 2021-03-09T08:56:36.278774+0000 mgr.y (mgr.14116) 14 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:56:38.163 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:38.156+0000 7f8ef80e5700 1 -- 172.21.15.5:0/2460155537 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f8ef0108f60 con 0x7f8ef00f9030 2021-03-09T08:56:38.864 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:38.855+0000 7f8eee7fc700 1 -- 172.21.15.5:0/2460155537 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mgrmap(e 9) v1 ==== 36073+0+0 (secure 0 0 0) 0x7f8ee0020030 con 0x7f8ef00f9030 2021-03-09T08:56:38.865 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:38.859+0000 7f8eee7fc700 1 -- 172.21.15.5:0/2460155537 <== mon.0 v2:172.21.15.5:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f8ee0041390 con 0x7f8ef00f9030 2021-03-09T08:56:38.869 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:38.863+0000 7f8ef80e5700 1 -- 172.21.15.5:0/2460155537 >> v2:172.21.15.5:6800/4233713404 conn(0x7f8edc02fde0 msgr2=0x7f8edc0322a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:38.871 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:38.863+0000 7f8ef80e5700 1 --2- 172.21.15.5:0/2460155537 >> v2:172.21.15.5:6800/4233713404 conn(0x7f8edc02fde0 0x7f8edc0322a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f8ee40096a0 tx=0x7f8ee4006b40).stop 2021-03-09T08:56:38.871 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:38.863+0000 7f8ef80e5700 1 -- 172.21.15.5:0/2460155537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef00f9030 msgr2=0x7f8ef0107170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:38.871 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:38.863+0000 7f8ef80e5700 1 --2- 172.21.15.5:0/2460155537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef00f9030 0x7f8ef0107170 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f8ee0006b80 tx=0x7f8ee00092d0).stop 2021-03-09T08:56:38.871 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:38.863+0000 7f8ef80e5700 1 -- 172.21.15.5:0/2460155537 shutdown_connections 2021-03-09T08:56:38.871 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:38.863+0000 7f8ef80e5700 1 --2- 172.21.15.5:0/2460155537 >> v2:172.21.15.5:6800/4233713404 conn(0x7f8edc02fde0 0x7f8edc0322a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:38.872 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:38.863+0000 7f8ef80e5700 1 --2- 172.21.15.5:0/2460155537 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef00f9030 0x7f8ef0107170 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:38.872 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:38.863+0000 7f8ef80e5700 1 -- 172.21.15.5:0/2460155537 >> 172.21.15.5:0/2460155537 conn(0x7f8ef00f3960 msgr2=0x7f8ef00fbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:38.872 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:38.863+0000 7f8ef80e5700 1 -- 172.21.15.5:0/2460155537 shutdown_connections 2021-03-09T08:56:38.872 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:38.863+0000 7f8ef80e5700 1 -- 172.21.15.5:0/2460155537 wait complete. 2021-03-09T08:56:39.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:38 smithi005 bash[12738]: audit 2021-03-09T08:56:38.162978+0000 mon.a (mon.0) 62 : audit [INF] from='client.? 172.21.15.5:0/2460155537' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2021-03-09T08:56:39.119 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:38 smithi005 bash[12977]: ignoring --setuser ceph since I am not root 2021-03-09T08:56:39.119 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:38 smithi005 bash[12977]: ignoring --setgroup ceph since I am not root 2021-03-09T08:56:39.584 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 mgr stat 2021-03-09T08:56:40.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:39 smithi005 bash[12738]: audit 2021-03-09T08:56:38.864320+0000 mon.a (mon.0) 63 : audit [INF] from='client.? 172.21.15.5:0/2460155537' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2021-03-09T08:56:40.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:39 smithi005 bash[12738]: cluster 2021-03-09T08:56:38.864403+0000 mon.a (mon.0) 64 : cluster [DBG] mgrmap e9: y(active, since 24s) 2021-03-09T08:56:40.797 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.791+0000 7fb8c9b7b700 1 Processor -- start 2021-03-09T08:56:40.797 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.791+0000 7fb8c9b7b700 1 -- start start 2021-03-09T08:56:40.797 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.791+0000 7fb8c9b7b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8c40f9e20 0x7fb8c40fa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:40.799 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.791+0000 7fb8c9b7b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c40fa730 con 0x7fb8c40f9e20 2021-03-09T08:56:40.799 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.791+0000 7fb8c8b79700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8c40f9e20 0x7fb8c40fa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:40.799 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c8b79700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8c40f9e20 0x7fb8c40fa240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46422/0 (socket says 172.21.15.5:46422) 2021-03-09T08:56:40.800 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c8b79700 1 -- 172.21.15.5:0/1251377827 learned_addr learned my addr 172.21.15.5:0/1251377827 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:40.800 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c8b79700 1 -- 172.21.15.5:0/1251377827 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c40fa870 con 0x7fb8c40f9e20 2021-03-09T08:56:40.800 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c8b79700 1 --2- 172.21.15.5:0/1251377827 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8c40f9e20 0x7fb8c40fa240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fb8b400b950 tx=0x7fb8b4006ee0).ready entity=mon.0 client_cookie=2bfc13013497b1b4 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:40.801 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c37fe700 1 -- 172.21.15.5:0/1251377827 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8b4009840 con 0x7fb8c40f9e20 2021-03-09T08:56:40.801 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c37fe700 1 -- 172.21.15.5:0/1251377827 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7fb8b4009e20 con 0x7fb8c40f9e20 2021-03-09T08:56:40.801 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c37fe700 1 -- 172.21.15.5:0/1251377827 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8b4011d10 con 0x7fb8c40f9e20 2021-03-09T08:56:40.802 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c9b7b700 1 -- 172.21.15.5:0/1251377827 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8c40f9e20 msgr2=0x7fb8c40fa240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:40.802 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c9b7b700 1 --2- 172.21.15.5:0/1251377827 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8c40f9e20 0x7fb8c40fa240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fb8b400b950 tx=0x7fb8b4006ee0).stop 2021-03-09T08:56:40.805 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c9b7b700 1 -- 172.21.15.5:0/1251377827 shutdown_connections 2021-03-09T08:56:40.805 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c9b7b700 1 --2- 172.21.15.5:0/1251377827 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8c40f9e20 0x7fb8c40fa240 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:40.805 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c9b7b700 1 -- 172.21.15.5:0/1251377827 >> 172.21.15.5:0/1251377827 conn(0x7fb8c40f37a0 msgr2=0x7fb8c40f5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:40.805 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c9b7b700 1 -- 172.21.15.5:0/1251377827 shutdown_connections 2021-03-09T08:56:40.806 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c9b7b700 1 -- 172.21.15.5:0/1251377827 wait complete. 2021-03-09T08:56:40.806 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c9b7b700 1 Processor -- start 2021-03-09T08:56:40.806 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c9b7b700 1 -- start start 2021-03-09T08:56:40.806 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c9b7b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8c40f9e20 0x7fb8c4105ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:40.807 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c9b7b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c41063b0 con 0x7fb8c40f9e20 2021-03-09T08:56:40.807 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c8b79700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8c40f9e20 0x7fb8c4105ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:40.807 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c8b79700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8c40f9e20 0x7fb8c4105ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46424/0 (socket says 172.21.15.5:46424) 2021-03-09T08:56:40.807 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c8b79700 1 -- 172.21.15.5:0/816105933 learned_addr learned my addr 172.21.15.5:0/816105933 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:40.807 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.795+0000 7fb8c8b79700 1 -- 172.21.15.5:0/816105933 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c41064f0 con 0x7fb8c40f9e20 2021-03-09T08:56:40.808 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.799+0000 7fb8c8b79700 1 --2- 172.21.15.5:0/816105933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8c40f9e20 0x7fb8c4105ec0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fb8b4009fd0 tx=0x7fb8b4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:40.808 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.799+0000 7fb8c17fa700 1 -- 172.21.15.5:0/816105933 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8b4009ca0 con 0x7fb8c40f9e20 2021-03-09T08:56:40.808 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.799+0000 7fb8c17fa700 1 -- 172.21.15.5:0/816105933 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7fb8b4011770 con 0x7fb8c40f9e20 2021-03-09T08:56:40.808 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.799+0000 7fb8c9b7b700 1 -- 172.21.15.5:0/816105933 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c4106740 con 0x7fb8c40f9e20 2021-03-09T08:56:40.809 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.799+0000 7fb8c17fa700 1 -- 172.21.15.5:0/816105933 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8b401bd20 con 0x7fb8c40f9e20 2021-03-09T08:56:40.809 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.799+0000 7fb8c9b7b700 1 -- 172.21.15.5:0/816105933 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c4107160 con 0x7fb8c40f9e20 2021-03-09T08:56:40.809 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.799+0000 7fb8c17fa700 1 -- 172.21.15.5:0/816105933 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 9) v1 ==== 36073+0+0 (secure 0 0 0) 0x7fb8b400f070 con 0x7fb8c40f9e20 2021-03-09T08:56:40.809 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.799+0000 7fb8c9b7b700 1 -- 172.21.15.5:0/816105933 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a4004da0 con 0x7fb8c40f9e20 2021-03-09T08:56:40.810 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.799+0000 7fb8c17fa700 1 --2- 172.21.15.5:0/816105933 >> v2:172.21.15.5:6800/4233713404 conn(0x7fb8b0032110 0x7fb8b00345d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:40.811 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.799+0000 7fb8c17fa700 1 -- 172.21.15.5:0/816105933 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7fb8b40416a0 con 0x7fb8c40f9e20 2021-03-09T08:56:40.811 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.799+0000 7fb8c3fff700 1 -- 172.21.15.5:0/816105933 >> v2:172.21.15.5:6800/4233713404 conn(0x7fb8b0032110 msgr2=0x7fb8b00345d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:6800/4233713404 2021-03-09T08:56:40.811 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.799+0000 7fb8c3fff700 1 --2- 172.21.15.5:0/816105933 >> v2:172.21.15.5:6800/4233713404 conn(0x7fb8b0032110 0x7fb8b00345d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:40.812 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:40.807+0000 7fb8c17fa700 1 -- 172.21.15.5:0/816105933 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7fb8b40182e0 con 0x7fb8c40f9e20 2021-03-09T08:56:41.006 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:41.003+0000 7fb8c3fff700 1 -- 172.21.15.5:0/816105933 >> v2:172.21.15.5:6800/4233713404 conn(0x7fb8b0032110 msgr2=0x7fb8b00345d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:6800/4233713404 2021-03-09T08:56:41.006 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:41.003+0000 7fb8c3fff700 1 --2- 172.21.15.5:0/816105933 >> v2:172.21.15.5:6800/4233713404 conn(0x7fb8b0032110 0x7fb8b00345d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T08:56:41.181 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:41.175+0000 7fb8c9b7b700 1 -- 172.21.15.5:0/816105933 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fb8a4005860 con 0x7fb8c40f9e20 2021-03-09T08:56:41.183 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:41.179+0000 7fb8c17fa700 1 -- 172.21.15.5:0/816105933 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fb8b4016070 con 0x7fb8c40f9e20 2021-03-09T08:56:41.183 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: { 2021-03-09T08:56:41.183 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 9, 2021-03-09T08:56:41.183 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "available": true, 2021-03-09T08:56:41.184 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "active_name": "y", 2021-03-09T08:56:41.184 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_standby": 0 2021-03-09T08:56:41.184 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: } 2021-03-09T08:56:41.186 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:41.179+0000 7fb8c9b7b700 1 -- 172.21.15.5:0/816105933 >> v2:172.21.15.5:6800/4233713404 conn(0x7fb8b0032110 msgr2=0x7fb8b00345d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:41.187 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:41.179+0000 7fb8c9b7b700 1 --2- 172.21.15.5:0/816105933 >> v2:172.21.15.5:6800/4233713404 conn(0x7fb8b0032110 0x7fb8b00345d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:41.187 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:41.179+0000 7fb8c9b7b700 1 -- 172.21.15.5:0/816105933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8c40f9e20 msgr2=0x7fb8c4105ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:41.187 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:41.179+0000 7fb8c9b7b700 1 --2- 172.21.15.5:0/816105933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8c40f9e20 0x7fb8c4105ec0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fb8b4009fd0 tx=0x7fb8b4006ee0).stop 2021-03-09T08:56:41.187 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:41.179+0000 7fb8c9b7b700 1 -- 172.21.15.5:0/816105933 shutdown_connections 2021-03-09T08:56:41.188 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:41.179+0000 7fb8c9b7b700 1 --2- 172.21.15.5:0/816105933 >> v2:172.21.15.5:6800/4233713404 conn(0x7fb8b0032110 0x7fb8b00345d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:41.188 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:41.179+0000 7fb8c9b7b700 1 --2- 172.21.15.5:0/816105933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8c40f9e20 0x7fb8c4105ec0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:41.188 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:41.179+0000 7fb8c9b7b700 1 -- 172.21.15.5:0/816105933 >> 172.21.15.5:0/816105933 conn(0x7fb8c40f37a0 msgr2=0x7fb8c40fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:41.188 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:41.179+0000 7fb8c9b7b700 1 -- 172.21.15.5:0/816105933 shutdown_connections 2021-03-09T08:56:41.189 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:41.179+0000 7fb8c9b7b700 1 -- 172.21.15.5:0/816105933 wait complete. 2021-03-09T08:56:41.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:41 smithi005 bash[12738]: audit 2021-03-09T08:56:41.179836+0000 mon.a (mon.0) 65 : audit [DBG] from='client.? 172.21.15.5:0/816105933' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-03-09T08:56:41.908 INFO:teuthology.orchestra.run.smithi005.stderr:Waiting for the mgr to restart... 2021-03-09T08:56:41.909 INFO:teuthology.orchestra.run.smithi005.stderr:Waiting for mgr epoch 9... 2021-03-09T08:56:41.909 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 tell mgr mgr_status 2021-03-09T08:56:43.078 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f4191fcb700 1 Processor -- start 2021-03-09T08:56:43.079 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f4191fcb700 1 -- start start 2021-03-09T08:56:43.080 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f4191fcb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f418c06fcf0 0x7f418c06e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:43.080 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f4191fcb700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c070110 con 0x7f418c06fcf0 2021-03-09T08:56:43.080 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f4190fc9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f418c06fcf0 0x7f418c06e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:43.081 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f4190fc9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f418c06fcf0 0x7f418c06e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46432/0 (socket says 172.21.15.5:46432) 2021-03-09T08:56:43.081 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f4190fc9700 1 -- 172.21.15.5:0/3974738211 learned_addr learned my addr 172.21.15.5:0/3974738211 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:43.081 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f4190fc9700 1 -- 172.21.15.5:0/3974738211 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c06e780 con 0x7f418c06fcf0 2021-03-09T08:56:43.081 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f4190fc9700 1 --2- 172.21.15.5:0/3974738211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f418c06fcf0 0x7f418c06e290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f418000b950 tx=0x7f4180006ee0).ready entity=mon.0 client_cookie=e52486c2ed887a8c server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:43.082 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f418b7fe700 1 -- 172.21.15.5:0/3974738211 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4180009840 con 0x7f418c06fcf0 2021-03-09T08:56:43.082 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f418b7fe700 1 -- 172.21.15.5:0/3974738211 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f4180009e20 con 0x7f418c06fcf0 2021-03-09T08:56:43.082 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f418b7fe700 1 -- 172.21.15.5:0/3974738211 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4180011d10 con 0x7f418c06fcf0 2021-03-09T08:56:43.085 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f4191fcb700 1 -- 172.21.15.5:0/3974738211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f418c06fcf0 msgr2=0x7f418c06e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:43.085 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f4191fcb700 1 --2- 172.21.15.5:0/3974738211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f418c06fcf0 0x7f418c06e290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f418000b950 tx=0x7f4180006ee0).stop 2021-03-09T08:56:43.086 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f4191fcb700 1 -- 172.21.15.5:0/3974738211 shutdown_connections 2021-03-09T08:56:43.086 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f4191fcb700 1 --2- 172.21.15.5:0/3974738211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f418c06fcf0 0x7f418c06e290 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:43.086 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f4191fcb700 1 -- 172.21.15.5:0/3974738211 >> 172.21.15.5:0/3974738211 conn(0x7f418c0f5120 msgr2=0x7f418c0f7560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:43.086 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f4191fcb700 1 -- 172.21.15.5:0/3974738211 shutdown_connections 2021-03-09T08:56:43.087 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.075+0000 7f4191fcb700 1 -- 172.21.15.5:0/3974738211 wait complete. 2021-03-09T08:56:43.087 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f4191fcb700 1 Processor -- start 2021-03-09T08:56:43.087 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f4191fcb700 1 -- start start 2021-03-09T08:56:43.087 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f4191fcb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f418c06fcf0 0x7f418c10a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:43.087 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f4191fcb700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c10a7f0 con 0x7f418c06fcf0 2021-03-09T08:56:43.088 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f4190fc9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f418c06fcf0 0x7f418c10a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:43.088 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f4190fc9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f418c06fcf0 0x7f418c10a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46434/0 (socket says 172.21.15.5:46434) 2021-03-09T08:56:43.088 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f4190fc9700 1 -- 172.21.15.5:0/1177710807 learned_addr learned my addr 172.21.15.5:0/1177710807 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:43.088 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f4190fc9700 1 -- 172.21.15.5:0/1177710807 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c10a930 con 0x7f418c06fcf0 2021-03-09T08:56:43.088 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f4190fc9700 1 --2- 172.21.15.5:0/1177710807 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f418c06fcf0 0x7f418c10a300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f4180009fd0 tx=0x7f4180006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:43.089 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f41897fa700 1 -- 172.21.15.5:0/1177710807 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f418001a650 con 0x7f418c06fcf0 2021-03-09T08:56:43.089 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f41897fa700 1 -- 172.21.15.5:0/1177710807 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f418001ac30 con 0x7f418c06fcf0 2021-03-09T08:56:43.089 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f4191fcb700 1 -- 172.21.15.5:0/1177710807 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418c10ab80 con 0x7f418c06fcf0 2021-03-09T08:56:43.089 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f4191fcb700 1 -- 172.21.15.5:0/1177710807 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418c10b6e0 con 0x7f418c06fcf0 2021-03-09T08:56:43.090 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f41897fa700 1 -- 172.21.15.5:0/1177710807 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f418001bbb0 con 0x7f418c06fcf0 2021-03-09T08:56:43.090 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f41897fa700 1 -- 172.21.15.5:0/1177710807 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 9) v1 ==== 36073+0+0 (secure 0 0 0) 0x7f418000f070 con 0x7f418c06fcf0 2021-03-09T08:56:43.090 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f41897fa700 1 --2- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4233713404 conn(0x7f417002fe70 0x7f4170032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:43.090 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f41897fa700 1 -- 172.21.15.5:0/1177710807 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f4180042260 con 0x7f418c06fcf0 2021-03-09T08:56:43.090 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f4191fcb700 1 -- 172.21.15.5:0/1177710807 --> v2:172.21.15.5:6800/4233713404 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f416c000d10 con 0x7f417002fe70 2021-03-09T08:56:43.091 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f418bfff700 1 -- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4233713404 conn(0x7f417002fe70 msgr2=0x7f4170032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:6800/4233713404 2021-03-09T08:56:43.091 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.079+0000 7f418bfff700 1 --2- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4233713404 conn(0x7f417002fe70 0x7f4170032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:43.286 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.283+0000 7f418bfff700 1 -- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4233713404 conn(0x7f417002fe70 msgr2=0x7f4170032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:6800/4233713404 2021-03-09T08:56:43.286 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.283+0000 7f418bfff700 1 --2- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4233713404 conn(0x7f417002fe70 0x7f4170032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T08:56:43.690 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.687+0000 7f418bfff700 1 -- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4233713404 conn(0x7f417002fe70 msgr2=0x7f4170032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:6800/4233713404 2021-03-09T08:56:43.690 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:43.687+0000 7f418bfff700 1 --2- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4233713404 conn(0x7f417002fe70 0x7f4170032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-03-09T08:56:44.494 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:44.491+0000 7f418bfff700 1 -- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4233713404 conn(0x7f417002fe70 msgr2=0x7f4170032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:6800/4233713404 2021-03-09T08:56:44.494 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:44.491+0000 7f418bfff700 1 --2- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4233713404 conn(0x7f417002fe70 0x7f4170032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-03-09T08:56:45.270 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:45.267+0000 7f41897fa700 1 -- 172.21.15.5:0/1177710807 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mgrmap(e 10) v1 ==== 35958+0+0 (secure 0 0 0) 0x7f418001ada0 con 0x7f418c06fcf0 2021-03-09T08:56:45.270 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:45.267+0000 7f41897fa700 1 -- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4233713404 conn(0x7f417002fe70 msgr2=0x7f4170032330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:45.271 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:45.267+0000 7f41897fa700 1 --2- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4233713404 conn(0x7f417002fe70 0x7f4170032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:45.869 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:56:45 smithi005 bash[12977]: Warning: Permanently added 'smithi005,172.21.15.5' (ECDSA) to the list of known hosts. 2021-03-09T08:56:46.274 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.267+0000 7f41897fa700 1 -- 172.21.15.5:0/1177710807 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mgrmap(e 11) v1 ==== 35993+0+0 (secure 0 0 0) 0x7f4180025460 con 0x7f418c06fcf0 2021-03-09T08:56:46.274 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.267+0000 7f41897fa700 1 --2- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4269212764 conn(0x7f41700337d0 0x7f4170035bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:46.275 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.271+0000 7f41897fa700 1 -- 172.21.15.5:0/1177710807 --> v2:172.21.15.5:6800/4269212764 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f416c000d10 con 0x7f41700337d0 2021-03-09T08:56:46.275 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.271+0000 7f418bfff700 1 --2- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4269212764 conn(0x7f41700337d0 0x7f4170035bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:46.276 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.271+0000 7f418bfff700 1 --2- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4269212764 conn(0x7f41700337d0 0x7f4170035bc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f41780036d0 tx=0x7f4178006d60).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:46.278 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.275+0000 7f41897fa700 1 -- 172.21.15.5:0/1177710807 <== mgr.14144 v2:172.21.15.5:6800/4269212764 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f416c000d10 con 0x7f41700337d0 2021-03-09T08:56:46.291 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.287+0000 7f4191fcb700 1 -- 172.21.15.5:0/1177710807 --> v2:172.21.15.5:6800/4269212764 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f416c002430 con 0x7f41700337d0 2021-03-09T08:56:46.292 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.287+0000 7f41897fa700 1 -- 172.21.15.5:0/1177710807 <== mgr.14144 v2:172.21.15.5:6800/4269212764 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f416c002430 con 0x7f41700337d0 2021-03-09T08:56:46.292 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: { 2021-03-09T08:56:46.292 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2021-03-09T08:56:46.293 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "initialized": true 2021-03-09T08:56:46.293 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: } 2021-03-09T08:56:46.295 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.287+0000 7f4191fcb700 1 -- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4269212764 conn(0x7f41700337d0 msgr2=0x7f4170035bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:46.295 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.287+0000 7f4191fcb700 1 --2- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4269212764 conn(0x7f41700337d0 0x7f4170035bc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f41780036d0 tx=0x7f4178006d60).stop 2021-03-09T08:56:46.295 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.287+0000 7f4191fcb700 1 -- 172.21.15.5:0/1177710807 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f418c06fcf0 msgr2=0x7f418c10a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:46.296 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.287+0000 7f4191fcb700 1 --2- 172.21.15.5:0/1177710807 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f418c06fcf0 0x7f418c10a300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f4180009fd0 tx=0x7f4180006ee0).stop 2021-03-09T08:56:46.296 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.287+0000 7f4191fcb700 1 -- 172.21.15.5:0/1177710807 shutdown_connections 2021-03-09T08:56:46.296 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.287+0000 7f4191fcb700 1 --2- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4269212764 conn(0x7f41700337d0 0x7f4170035bc0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:46.296 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.287+0000 7f4191fcb700 1 --2- 172.21.15.5:0/1177710807 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f418c06fcf0 0x7f418c10a300 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:46.297 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.287+0000 7f4191fcb700 1 --2- 172.21.15.5:0/1177710807 >> v2:172.21.15.5:6800/4233713404 conn(0x7f417002fe70 0x7f4170032330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:46.297 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.287+0000 7f4191fcb700 1 -- 172.21.15.5:0/1177710807 >> 172.21.15.5:0/1177710807 conn(0x7f418c0f5120 msgr2=0x7f418c0f74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:46.297 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.287+0000 7f4191fcb700 1 -- 172.21.15.5:0/1177710807 shutdown_connections 2021-03-09T08:56:46.297 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:46.287+0000 7f4191fcb700 1 -- 172.21.15.5:0/1177710807 wait complete. 2021-03-09T08:56:46.488 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:46 smithi005 bash[12738]: cluster 2021-03-09T08:56:45.213815+0000 mon.a (mon.0) 66 : cluster [INF] Active manager daemon y restarted 2021-03-09T08:56:46.489 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:46 smithi005 bash[12738]: cluster 2021-03-09T08:56:45.214527+0000 mon.a (mon.0) 67 : cluster [INF] Activating manager daemon y 2021-03-09T08:56:46.489 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:46 smithi005 bash[12738]: cluster 2021-03-09T08:56:45.217462+0000 mon.a (mon.0) 68 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2021-03-09T08:56:46.489 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:46 smithi005 bash[12738]: cluster 2021-03-09T08:56:45.268393+0000 mon.a (mon.0) 69 : cluster [DBG] mgrmap e10: y(active, starting, since 0.0540487s) 2021-03-09T08:56:46.489 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:46 smithi005 bash[12738]: audit 2021-03-09T08:56:45.272687+0000 mon.a (mon.0) 70 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-03-09T08:56:46.490 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:46 smithi005 bash[12738]: audit 2021-03-09T08:56:45.274474+0000 mon.a (mon.0) 71 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-03-09T08:56:46.490 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:46 smithi005 bash[12738]: audit 2021-03-09T08:56:45.275778+0000 mon.a (mon.0) 72 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-03-09T08:56:46.490 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:46 smithi005 bash[12738]: audit 2021-03-09T08:56:45.276212+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-03-09T08:56:46.490 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:46 smithi005 bash[12738]: audit 2021-03-09T08:56:45.276623+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-03-09T08:56:46.491 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:46 smithi005 bash[12738]: cluster 2021-03-09T08:56:45.280932+0000 mon.a (mon.0) 75 : cluster [INF] Manager daemon y is now available 2021-03-09T08:56:46.491 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:46 smithi005 bash[12738]: audit 2021-03-09T08:56:45.434593+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:56:46.491 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:46 smithi005 bash[12738]: audit 2021-03-09T08:56:45.478540+0000 mon.a (mon.0) 77 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-09T08:56:46.491 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:46 smithi005 bash[12738]: audit 2021-03-09T08:56:45.482320+0000 mon.a (mon.0) 78 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-09T08:56:47.040 INFO:teuthology.orchestra.run.smithi005.stderr:mgr epoch 9 is available 2021-03-09T08:56:47.040 INFO:teuthology.orchestra.run.smithi005.stderr:Generating a dashboard self-signed certificate... 2021-03-09T08:56:47.041 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 dashboard create-self-signed-cert 2021-03-09T08:56:47.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:47 smithi005 bash[12738]: audit 2021-03-09T08:56:46.274957+0000 mgr.y (mgr.14144) 1 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-03-09T08:56:47.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:47 smithi005 bash[12738]: cluster 2021-03-09T08:56:46.275624+0000 mon.a (mon.0) 79 : cluster [DBG] mgrmap e11: y(active, since 1.06126s) 2021-03-09T08:56:47.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:47 smithi005 bash[12738]: audit 2021-03-09T08:56:46.289482+0000 mgr.y (mgr.14144) 2 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-03-09T08:56:48.200 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.195+0000 7fbf8e739700 1 Processor -- start 2021-03-09T08:56:48.200 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.195+0000 7fbf8e739700 1 -- start start 2021-03-09T08:56:48.201 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.195+0000 7fbf8e739700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf880742d0 0x7fbf88102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:48.202 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.195+0000 7fbf8e739700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf881031e0 con 0x7fbf880742d0 2021-03-09T08:56:48.202 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.195+0000 7fbf87fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf880742d0 0x7fbf88102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:48.202 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.195+0000 7fbf87fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf880742d0 0x7fbf88102cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46470/0 (socket says 172.21.15.5:46470) 2021-03-09T08:56:48.202 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.195+0000 7fbf87fff700 1 -- 172.21.15.5:0/1254037281 learned_addr learned my addr 172.21.15.5:0/1254037281 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:48.203 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.195+0000 7fbf87fff700 1 -- 172.21.15.5:0/1254037281 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf88103320 con 0x7fbf880742d0 2021-03-09T08:56:48.204 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.195+0000 7fbf87fff700 1 --2- 172.21.15.5:0/1254037281 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf880742d0 0x7fbf88102cf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fbf7c00b950 tx=0x7fbf7c006ee0).ready entity=mon.0 client_cookie=656493e9c6c4ee3e server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:48.204 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.199+0000 7fbf86ffd700 1 -- 172.21.15.5:0/1254037281 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf7c009840 con 0x7fbf880742d0 2021-03-09T08:56:48.204 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.199+0000 7fbf86ffd700 1 -- 172.21.15.5:0/1254037281 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7fbf7c009e20 con 0x7fbf880742d0 2021-03-09T08:56:48.204 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.199+0000 7fbf86ffd700 1 -- 172.21.15.5:0/1254037281 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf7c011d10 con 0x7fbf880742d0 2021-03-09T08:56:48.205 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.199+0000 7fbf8e739700 1 -- 172.21.15.5:0/1254037281 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf880742d0 msgr2=0x7fbf88102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:48.205 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.199+0000 7fbf8e739700 1 --2- 172.21.15.5:0/1254037281 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf880742d0 0x7fbf88102cf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fbf7c00b950 tx=0x7fbf7c006ee0).stop 2021-03-09T08:56:48.207 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.199+0000 7fbf8e739700 1 -- 172.21.15.5:0/1254037281 shutdown_connections 2021-03-09T08:56:48.207 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.199+0000 7fbf8e739700 1 --2- 172.21.15.5:0/1254037281 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf880742d0 0x7fbf88102cf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:48.208 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.199+0000 7fbf8e739700 1 -- 172.21.15.5:0/1254037281 >> 172.21.15.5:0/1254037281 conn(0x7fbf8806f450 msgr2=0x7fbf88071870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:48.208 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.199+0000 7fbf8e739700 1 -- 172.21.15.5:0/1254037281 shutdown_connections 2021-03-09T08:56:48.208 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.199+0000 7fbf8e739700 1 -- 172.21.15.5:0/1254037281 wait complete. 2021-03-09T08:56:48.208 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.199+0000 7fbf8e739700 1 Processor -- start 2021-03-09T08:56:48.208 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.199+0000 7fbf8e739700 1 -- start start 2021-03-09T08:56:48.209 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.199+0000 7fbf8e739700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf880742d0 0x7fbf8810e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:48.209 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.199+0000 7fbf8e739700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8810ed40 con 0x7fbf880742d0 2021-03-09T08:56:48.209 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.199+0000 7fbf87fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf880742d0 0x7fbf8810e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:48.209 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.203+0000 7fbf87fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf880742d0 0x7fbf8810e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46472/0 (socket says 172.21.15.5:46472) 2021-03-09T08:56:48.210 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.203+0000 7fbf87fff700 1 -- 172.21.15.5:0/474500451 learned_addr learned my addr 172.21.15.5:0/474500451 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:48.210 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.203+0000 7fbf87fff700 1 -- 172.21.15.5:0/474500451 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8810ee80 con 0x7fbf880742d0 2021-03-09T08:56:48.210 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.203+0000 7fbf87fff700 1 --2- 172.21.15.5:0/474500451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf880742d0 0x7fbf8810e850 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fbf7c000c00 tx=0x7fbf7c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:48.210 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.203+0000 7fbf84ff9700 1 -- 172.21.15.5:0/474500451 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf7c009ca0 con 0x7fbf880742d0 2021-03-09T08:56:48.210 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.203+0000 7fbf84ff9700 1 -- 172.21.15.5:0/474500451 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7fbf7c010f00 con 0x7fbf880742d0 2021-03-09T08:56:48.212 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.203+0000 7fbf84ff9700 1 -- 172.21.15.5:0/474500451 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf7c01bcc0 con 0x7fbf880742d0 2021-03-09T08:56:48.212 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.203+0000 7fbf8e739700 1 -- 172.21.15.5:0/474500451 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf8810f0d0 con 0x7fbf880742d0 2021-03-09T08:56:48.212 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.203+0000 7fbf8e739700 1 -- 172.21.15.5:0/474500451 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf8810fc30 con 0x7fbf880742d0 2021-03-09T08:56:48.213 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.203+0000 7fbf84ff9700 1 -- 172.21.15.5:0/474500451 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 12) v1 ==== 36073+0+0 (secure 0 0 0) 0x7fbf7c00f070 con 0x7fbf880742d0 2021-03-09T08:56:48.213 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.203+0000 7fbf84ff9700 1 --2- 172.21.15.5:0/474500451 >> v2:172.21.15.5:6800/4269212764 conn(0x7fbf7402fe70 0x7fbf74032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:48.213 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.203+0000 7fbf84ff9700 1 -- 172.21.15.5:0/474500451 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7fbf7c041a20 con 0x7fbf880742d0 2021-03-09T08:56:48.213 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.203+0000 7fbf877fe700 1 --2- 172.21.15.5:0/474500451 >> v2:172.21.15.5:6800/4269212764 conn(0x7fbf7402fe70 0x7fbf74032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:48.213 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.203+0000 7fbf877fe700 1 --2- 172.21.15.5:0/474500451 >> v2:172.21.15.5:6800/4269212764 conn(0x7fbf7402fe70 0x7fbf74032330 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbf800096a0 tx=0x7fbf80006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:48.214 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.203+0000 7fbf6e7fc700 1 -- 172.21.15.5:0/474500451 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf60004da0 con 0x7fbf880742d0 2021-03-09T08:56:48.216 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.211+0000 7fbf84ff9700 1 -- 172.21.15.5:0/474500451 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7fbf7c016070 con 0x7fbf880742d0 2021-03-09T08:56:48.526 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.523+0000 7fbf6e7fc700 1 -- 172.21.15.5:0/474500451 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf60002870 con 0x7fbf7402fe70 2021-03-09T08:56:48.647 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.643+0000 7fbf84ff9700 1 -- 172.21.15.5:0/474500451 <== mgr.14144 v2:172.21.15.5:6800/4269212764 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fbf60002870 con 0x7fbf7402fe70 2021-03-09T08:56:48.647 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: Self-signed certificate created 2021-03-09T08:56:48.650 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.643+0000 7fbf6e7fc700 1 -- 172.21.15.5:0/474500451 >> v2:172.21.15.5:6800/4269212764 conn(0x7fbf7402fe70 msgr2=0x7fbf74032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:48.650 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.643+0000 7fbf6e7fc700 1 --2- 172.21.15.5:0/474500451 >> v2:172.21.15.5:6800/4269212764 conn(0x7fbf7402fe70 0x7fbf74032330 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbf800096a0 tx=0x7fbf80006b40).stop 2021-03-09T08:56:48.651 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.643+0000 7fbf6e7fc700 1 -- 172.21.15.5:0/474500451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf880742d0 msgr2=0x7fbf8810e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:48.651 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.643+0000 7fbf6e7fc700 1 --2- 172.21.15.5:0/474500451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf880742d0 0x7fbf8810e850 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fbf7c000c00 tx=0x7fbf7c006ee0).stop 2021-03-09T08:56:48.651 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.643+0000 7fbf6e7fc700 1 -- 172.21.15.5:0/474500451 shutdown_connections 2021-03-09T08:56:48.651 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.643+0000 7fbf6e7fc700 1 --2- 172.21.15.5:0/474500451 >> v2:172.21.15.5:6800/4269212764 conn(0x7fbf7402fe70 0x7fbf74032330 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:48.652 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.643+0000 7fbf6e7fc700 1 --2- 172.21.15.5:0/474500451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf880742d0 0x7fbf8810e850 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:48.652 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.643+0000 7fbf6e7fc700 1 -- 172.21.15.5:0/474500451 >> 172.21.15.5:0/474500451 conn(0x7fbf8806f450 msgr2=0x7fbf880727f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:48.652 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.643+0000 7fbf6e7fc700 1 -- 172.21.15.5:0/474500451 shutdown_connections 2021-03-09T08:56:48.652 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:48.647+0000 7fbf6e7fc700 1 -- 172.21.15.5:0/474500451 wait complete. 2021-03-09T08:56:49.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:48 smithi005 bash[12738]: cluster 2021-03-09T08:56:47.860385+0000 mon.a (mon.0) 80 : cluster [DBG] mgrmap e12: y(active, since 2s) 2021-03-09T08:56:49.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:48 smithi005 bash[12738]: audit 2021-03-09T08:56:48.641358+0000 mon.a (mon.0) 81 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:56:49.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:48 smithi005 bash[12738]: audit 2021-03-09T08:56:48.643709+0000 mon.a (mon.0) 82 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:56:49.364 INFO:teuthology.orchestra.run.smithi005.stderr:Creating initial admin user... 2021-03-09T08:56:49.365 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpe_8y2tj3:/tmp/dashboard.pw:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2021-03-09T08:56:50.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:49 smithi005 bash[12738]: audit 2021-03-09T08:56:48.525145+0000 mgr.y (mgr.14144) 3 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:56:50.555 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.551+0000 7fcfb5ebd700 1 Processor -- start 2021-03-09T08:56:50.556 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.551+0000 7fcfb5ebd700 1 -- start start 2021-03-09T08:56:50.556 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.551+0000 7fcfb5ebd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcfb00f7fc0 0x7fcfb00f83e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:50.556 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.551+0000 7fcfb5ebd700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb00f88d0 con 0x7fcfb00f7fc0 2021-03-09T08:56:50.557 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.551+0000 7fcfb4ebb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcfb00f7fc0 0x7fcfb00f83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:50.562 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.551+0000 7fcfb4ebb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcfb00f7fc0 0x7fcfb00f83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46476/0 (socket says 172.21.15.5:46476) 2021-03-09T08:56:50.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.551+0000 7fcfb4ebb700 1 -- 172.21.15.5:0/2754897773 learned_addr learned my addr 172.21.15.5:0/2754897773 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:50.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.551+0000 7fcfb4ebb700 1 -- 172.21.15.5:0/2754897773 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb00f8a10 con 0x7fcfb00f7fc0 2021-03-09T08:56:50.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.551+0000 7fcfb4ebb700 1 --2- 172.21.15.5:0/2754897773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcfb00f7fc0 0x7fcfb00f83e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fcfa400b950 tx=0x7fcfa4006ee0).ready entity=mon.0 client_cookie=cf976eacff6bea6f server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:50.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.551+0000 7fcfaf7fe700 1 -- 172.21.15.5:0/2754897773 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcfa4009840 con 0x7fcfb00f7fc0 2021-03-09T08:56:50.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.551+0000 7fcfaf7fe700 1 -- 172.21.15.5:0/2754897773 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7fcfa4009e20 con 0x7fcfb00f7fc0 2021-03-09T08:56:50.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.551+0000 7fcfaf7fe700 1 -- 172.21.15.5:0/2754897773 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcfa4011d10 con 0x7fcfb00f7fc0 2021-03-09T08:56:50.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.555+0000 7fcfb5ebd700 1 -- 172.21.15.5:0/2754897773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcfb00f7fc0 msgr2=0x7fcfb00f83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:50.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.555+0000 7fcfb5ebd700 1 --2- 172.21.15.5:0/2754897773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcfb00f7fc0 0x7fcfb00f83e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fcfa400b950 tx=0x7fcfa4006ee0).stop 2021-03-09T08:56:50.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfb5ebd700 1 -- 172.21.15.5:0/2754897773 shutdown_connections 2021-03-09T08:56:50.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfb5ebd700 1 --2- 172.21.15.5:0/2754897773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcfb00f7fc0 0x7fcfb00f83e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:50.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfb5ebd700 1 -- 172.21.15.5:0/2754897773 >> 172.21.15.5:0/2754897773 conn(0x7fcfb00f37e0 msgr2=0x7fcfb00f5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:50.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfb5ebd700 1 -- 172.21.15.5:0/2754897773 shutdown_connections 2021-03-09T08:56:50.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfb5ebd700 1 -- 172.21.15.5:0/2754897773 wait complete. 2021-03-09T08:56:50.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfb5ebd700 1 Processor -- start 2021-03-09T08:56:50.567 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfb5ebd700 1 -- start start 2021-03-09T08:56:50.567 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfb5ebd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcfb00f7fc0 0x7fcfb0105ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:50.567 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfb5ebd700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb01063e0 con 0x7fcfb00f7fc0 2021-03-09T08:56:50.567 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfb4ebb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcfb00f7fc0 0x7fcfb0105ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:50.567 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfb4ebb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcfb00f7fc0 0x7fcfb0105ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46478/0 (socket says 172.21.15.5:46478) 2021-03-09T08:56:50.568 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfb4ebb700 1 -- 172.21.15.5:0/592373825 learned_addr learned my addr 172.21.15.5:0/592373825 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:50.568 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfb4ebb700 1 -- 172.21.15.5:0/592373825 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb0106520 con 0x7fcfb00f7fc0 2021-03-09T08:56:50.568 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfb4ebb700 1 --2- 172.21.15.5:0/592373825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcfb00f7fc0 0x7fcfb0105ef0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fcfa4006a40 tx=0x7fcfa4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:50.568 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfad7fa700 1 -- 172.21.15.5:0/592373825 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcfa401f070 con 0x7fcfb00f7fc0 2021-03-09T08:56:50.569 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfad7fa700 1 -- 172.21.15.5:0/592373825 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7fcfa401aad0 con 0x7fcfb00f7fc0 2021-03-09T08:56:50.569 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.559+0000 7fcfad7fa700 1 -- 172.21.15.5:0/592373825 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcfa401bb00 con 0x7fcfb00f7fc0 2021-03-09T08:56:50.569 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.563+0000 7fcfb5ebd700 1 -- 172.21.15.5:0/592373825 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb0106770 con 0x7fcfb00f7fc0 2021-03-09T08:56:50.570 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.563+0000 7fcfb5ebd700 1 -- 172.21.15.5:0/592373825 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb01072d0 con 0x7fcfb00f7fc0 2021-03-09T08:56:50.570 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.563+0000 7fcfad7fa700 1 -- 172.21.15.5:0/592373825 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 12) v1 ==== 36073+0+0 (secure 0 0 0) 0x7fcfa400f070 con 0x7fcfb00f7fc0 2021-03-09T08:56:50.571 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.563+0000 7fcfad7fa700 1 --2- 172.21.15.5:0/592373825 >> v2:172.21.15.5:6800/4269212764 conn(0x7fcf94032070 0x7fcf94034530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:50.571 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.563+0000 7fcfad7fa700 1 -- 172.21.15.5:0/592373825 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7fcfa4041330 con 0x7fcfb00f7fc0 2021-03-09T08:56:50.571 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.563+0000 7fcfaffff700 1 --2- 172.21.15.5:0/592373825 >> v2:172.21.15.5:6800/4269212764 conn(0x7fcf94032070 0x7fcf94034530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:50.572 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.563+0000 7fcfb5ebd700 1 -- 172.21.15.5:0/592373825 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb00589f0 con 0x7fcfb00f7fc0 2021-03-09T08:56:50.572 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.563+0000 7fcfaffff700 1 --2- 172.21.15.5:0/592373825 >> v2:172.21.15.5:6800/4269212764 conn(0x7fcf94032070 0x7fcf94034530 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fcf9c0096a0 tx=0x7fcf9c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:50.572 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.567+0000 7fcfad7fa700 1 -- 172.21.15.5:0/592373825 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7fcfa4016070 con 0x7fcfb00f7fc0 2021-03-09T08:56:50.904 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:50.899+0000 7fcfb5ebd700 1 -- 172.21.15.5:0/592373825 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fcfb0058810 con 0x7fcf94032070 2021-03-09T08:56:51.165 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:51.159+0000 7fcfad7fa700 1 -- 172.21.15.5:0/592373825 <== mgr.14144 v2:172.21.15.5:6800/4269212764 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fcfb0058810 con 0x7fcf94032070 2021-03-09T08:56:51.165 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$hMpAjBuZYWm4kXDO/ROUP.0r7MQx.PZ3YhUBUuIRRJyGju3ZbnhRi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1615280211, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2021-03-09T08:56:51.169 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:51.163+0000 7fcfb5ebd700 1 -- 172.21.15.5:0/592373825 >> v2:172.21.15.5:6800/4269212764 conn(0x7fcf94032070 msgr2=0x7fcf94034530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:51.169 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:51.163+0000 7fcfb5ebd700 1 --2- 172.21.15.5:0/592373825 >> v2:172.21.15.5:6800/4269212764 conn(0x7fcf94032070 0x7fcf94034530 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fcf9c0096a0 tx=0x7fcf9c006b40).stop 2021-03-09T08:56:51.170 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:51.163+0000 7fcfb5ebd700 1 -- 172.21.15.5:0/592373825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcfb00f7fc0 msgr2=0x7fcfb0105ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:51.170 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:51.163+0000 7fcfb5ebd700 1 --2- 172.21.15.5:0/592373825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcfb00f7fc0 0x7fcfb0105ef0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fcfa4006a40 tx=0x7fcfa4006ee0).stop 2021-03-09T08:56:51.171 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:51.163+0000 7fcfb5ebd700 1 -- 172.21.15.5:0/592373825 shutdown_connections 2021-03-09T08:56:51.171 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:51.163+0000 7fcfb5ebd700 1 --2- 172.21.15.5:0/592373825 >> v2:172.21.15.5:6800/4269212764 conn(0x7fcf94032070 0x7fcf94034530 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:51.171 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:51.163+0000 7fcfb5ebd700 1 --2- 172.21.15.5:0/592373825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcfb00f7fc0 0x7fcfb0105ef0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:51.171 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:51.163+0000 7fcfb5ebd700 1 -- 172.21.15.5:0/592373825 >> 172.21.15.5:0/592373825 conn(0x7fcfb00f37e0 msgr2=0x7fcfb00fad40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:51.172 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:51.167+0000 7fcfb5ebd700 1 -- 172.21.15.5:0/592373825 shutdown_connections 2021-03-09T08:56:51.172 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:51.167+0000 7fcfb5ebd700 1 -- 172.21.15.5:0/592373825 wait complete. 2021-03-09T08:56:51.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:51 smithi005 bash[12738]: audit 2021-03-09T08:56:50.324857+0000 mon.a (mon.0) 83 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:56:51.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:51 smithi005 bash[12738]: audit 2021-03-09T08:56:50.903891+0000 mgr.y (mgr.14144) 4 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:56:51.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:51 smithi005 bash[12738]: audit 2021-03-09T08:56:51.162555+0000 mon.a (mon.0) 84 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:56:52.401 INFO:teuthology.orchestra.run.smithi005.stderr:Fetching dashboard port number... 2021-03-09T08:56:52.401 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 -e NODE_NAME=smithi005 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/252abbe0-80b5-11eb-907b-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmplnl3x1lr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy46n7d4a:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 config get mgr mgr/dashboard/ssl_server_port 2021-03-09T08:56:53.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:53 smithi005 bash[12738]: cluster 2021-03-09T08:56:52.167823+0000 mon.a (mon.0) 85 : cluster [DBG] mgrmap e13: y(active, since 6s) 2021-03-09T08:56:54.278 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.271+0000 7f898e7d1700 1 Processor -- start 2021-03-09T08:56:54.279 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.271+0000 7f898e7d1700 1 -- start start 2021-03-09T08:56:54.279 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.271+0000 7f898e7d1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89880f9940 0x7f89880f9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:54.279 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.275+0000 7f898e7d1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89880fa250 con 0x7f89880f9940 2021-03-09T08:56:54.279 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.275+0000 7f898d7cf700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89880f9940 0x7f89880f9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:54.281 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.275+0000 7f898d7cf700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89880f9940 0x7f89880f9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46486/0 (socket says 172.21.15.5:46486) 2021-03-09T08:56:54.282 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.275+0000 7f898d7cf700 1 -- 172.21.15.5:0/4197783602 learned_addr learned my addr 172.21.15.5:0/4197783602 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:54.282 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.275+0000 7f898d7cf700 1 -- 172.21.15.5:0/4197783602 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89880fa390 con 0x7f89880f9940 2021-03-09T08:56:54.282 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.275+0000 7f898d7cf700 1 --2- 172.21.15.5:0/4197783602 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89880f9940 0x7f89880f9d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f898400b950 tx=0x7f8984006ee0).ready entity=mon.0 client_cookie=8726f81d023ce58f server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:54.282 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.275+0000 7f8973fff700 1 -- 172.21.15.5:0/4197783602 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8984009840 con 0x7f89880f9940 2021-03-09T08:56:54.283 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.275+0000 7f8973fff700 1 -- 172.21.15.5:0/4197783602 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8984009e20 con 0x7f89880f9940 2021-03-09T08:56:54.283 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.275+0000 7f8973fff700 1 -- 172.21.15.5:0/4197783602 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8984011d10 con 0x7f89880f9940 2021-03-09T08:56:54.283 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.275+0000 7f898e7d1700 1 -- 172.21.15.5:0/4197783602 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89880f9940 msgr2=0x7f89880f9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:54.283 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.275+0000 7f898e7d1700 1 --2- 172.21.15.5:0/4197783602 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89880f9940 0x7f89880f9d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f898400b950 tx=0x7f8984006ee0).stop 2021-03-09T08:56:54.283 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.275+0000 7f898e7d1700 1 -- 172.21.15.5:0/4197783602 shutdown_connections 2021-03-09T08:56:54.284 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.275+0000 7f898e7d1700 1 --2- 172.21.15.5:0/4197783602 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89880f9940 0x7f89880f9d60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:54.284 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.275+0000 7f898e7d1700 1 -- 172.21.15.5:0/4197783602 >> 172.21.15.5:0/4197783602 conn(0x7f89880f51a0 msgr2=0x7f89880f75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:54.284 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.275+0000 7f898e7d1700 1 -- 172.21.15.5:0/4197783602 shutdown_connections 2021-03-09T08:56:54.284 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.275+0000 7f898e7d1700 1 -- 172.21.15.5:0/4197783602 wait complete. 2021-03-09T08:56:54.285 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.279+0000 7f898e7d1700 1 Processor -- start 2021-03-09T08:56:54.285 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.279+0000 7f898e7d1700 1 -- start start 2021-03-09T08:56:54.287 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.279+0000 7f898e7d1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89880f9940 0x7f8988108100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:54.287 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.279+0000 7f898e7d1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89881085f0 con 0x7f89880f9940 2021-03-09T08:56:54.288 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.279+0000 7f898d7cf700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89880f9940 0x7f8988108100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:54.288 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.279+0000 7f898d7cf700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89880f9940 0x7f8988108100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46488/0 (socket says 172.21.15.5:46488) 2021-03-09T08:56:54.288 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.279+0000 7f898d7cf700 1 -- 172.21.15.5:0/1632425372 learned_addr learned my addr 172.21.15.5:0/1632425372 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:54.288 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.279+0000 7f898d7cf700 1 -- 172.21.15.5:0/1632425372 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8988108730 con 0x7f89880f9940 2021-03-09T08:56:54.289 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.279+0000 7f898d7cf700 1 --2- 172.21.15.5:0/1632425372 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89880f9940 0x7f8988108100 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f8984006a40 tx=0x7f8984006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:54.289 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.279+0000 7f897a7fc700 1 -- 172.21.15.5:0/1632425372 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f898401f070 con 0x7f89880f9940 2021-03-09T08:56:54.289 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.279+0000 7f898e7d1700 1 -- 172.21.15.5:0/1632425372 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8988108980 con 0x7f89880f9940 2021-03-09T08:56:54.289 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.279+0000 7f898e7d1700 1 -- 172.21.15.5:0/1632425372 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89881094e0 con 0x7f89880f9940 2021-03-09T08:56:54.289 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.279+0000 7f897a7fc700 1 -- 172.21.15.5:0/1632425372 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f898401aad0 con 0x7f89880f9940 2021-03-09T08:56:54.290 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.279+0000 7f897a7fc700 1 -- 172.21.15.5:0/1632425372 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f898401bb00 con 0x7f89880f9940 2021-03-09T08:56:54.290 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.283+0000 7f898e7d1700 1 -- 172.21.15.5:0/1632425372 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89880589f0 con 0x7f89880f9940 2021-03-09T08:56:54.290 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.283+0000 7f897a7fc700 1 -- 172.21.15.5:0/1632425372 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 13) v1 ==== 36113+0+0 (secure 0 0 0) 0x7f898400f070 con 0x7f89880f9940 2021-03-09T08:56:54.290 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.283+0000 7f897a7fc700 1 --2- 172.21.15.5:0/1632425372 >> v2:172.21.15.5:6800/4269212764 conn(0x7f896c02ff00 0x7f896c0323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:54.290 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.283+0000 7f897a7fc700 1 -- 172.21.15.5:0/1632425372 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f8984041560 con 0x7f89880f9940 2021-03-09T08:56:54.291 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.283+0000 7f898cfce700 1 --2- 172.21.15.5:0/1632425372 >> v2:172.21.15.5:6800/4269212764 conn(0x7f896c02ff00 0x7f896c0323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:54.291 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.283+0000 7f898cfce700 1 --2- 172.21.15.5:0/1632425372 >> v2:172.21.15.5:6800/4269212764 conn(0x7f896c02ff00 0x7f896c0323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f897c0095f0 tx=0x7f897c006b30).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:54.293 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.287+0000 7f897a7fc700 1 -- 172.21.15.5:0/1632425372 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f8984022960 con 0x7f89880f9940 2021-03-09T08:56:54.592 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.587+0000 7f898e7d1700 1 -- 172.21.15.5:0/1632425372 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f8988045280 con 0x7f89880f9940 2021-03-09T08:56:54.593 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.587+0000 7f897a7fc700 1 -- 172.21.15.5:0/1632425372 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f8984016070 con 0x7f89880f9940 2021-03-09T08:56:54.593 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 8443 2021-03-09T08:56:54.595 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.591+0000 7f898e7d1700 1 -- 172.21.15.5:0/1632425372 >> v2:172.21.15.5:6800/4269212764 conn(0x7f896c02ff00 msgr2=0x7f896c0323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:54.596 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.591+0000 7f898e7d1700 1 --2- 172.21.15.5:0/1632425372 >> v2:172.21.15.5:6800/4269212764 conn(0x7f896c02ff00 0x7f896c0323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f897c0095f0 tx=0x7f897c006b30).stop 2021-03-09T08:56:54.596 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.591+0000 7f898e7d1700 1 -- 172.21.15.5:0/1632425372 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89880f9940 msgr2=0x7f8988108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:54.596 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.591+0000 7f898e7d1700 1 --2- 172.21.15.5:0/1632425372 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89880f9940 0x7f8988108100 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f8984006a40 tx=0x7f8984006ee0).stop 2021-03-09T08:56:54.597 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.591+0000 7f898e7d1700 1 -- 172.21.15.5:0/1632425372 shutdown_connections 2021-03-09T08:56:54.597 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.591+0000 7f898e7d1700 1 --2- 172.21.15.5:0/1632425372 >> v2:172.21.15.5:6800/4269212764 conn(0x7f896c02ff00 0x7f896c0323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:54.598 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.591+0000 7f898e7d1700 1 --2- 172.21.15.5:0/1632425372 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89880f9940 0x7f8988108100 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:54.598 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.591+0000 7f898e7d1700 1 -- 172.21.15.5:0/1632425372 >> 172.21.15.5:0/1632425372 conn(0x7f89880f51a0 msgr2=0x7f89880f6830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:54.598 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.591+0000 7f898e7d1700 1 -- 172.21.15.5:0/1632425372 shutdown_connections 2021-03-09T08:56:54.599 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2021-03-09T08:56:54.591+0000 7f898e7d1700 1 -- 172.21.15.5:0/1632425372 wait complete. 2021-03-09T08:56:55.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:56:54 smithi005 bash[12738]: audit 2021-03-09T08:56:54.591164+0000 mon.a (mon.0) 86 : audit [DBG] from='client.? 172.21.15.5:0/1632425372' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2021-03-09T08:56:55.294 INFO:teuthology.orchestra.run.smithi005.stderr:firewalld does not appear to be present 2021-03-09T08:56:55.295 INFO:teuthology.orchestra.run.smithi005.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2021-03-09T08:56:55.297 INFO:teuthology.orchestra.run.smithi005.stderr:Ceph Dashboard is now available at: 2021-03-09T08:56:55.297 INFO:teuthology.orchestra.run.smithi005.stderr: 2021-03-09T08:56:55.297 INFO:teuthology.orchestra.run.smithi005.stderr: URL: https://smithi005:8443/ 2021-03-09T08:56:55.298 INFO:teuthology.orchestra.run.smithi005.stderr: User: admin 2021-03-09T08:56:55.298 INFO:teuthology.orchestra.run.smithi005.stderr: Password: 9uz81kkfe6 2021-03-09T08:56:55.298 INFO:teuthology.orchestra.run.smithi005.stderr: 2021-03-09T08:56:55.298 INFO:teuthology.orchestra.run.smithi005.stderr:You can access the Ceph CLI with: 2021-03-09T08:56:55.299 INFO:teuthology.orchestra.run.smithi005.stderr: 2021-03-09T08:56:55.299 INFO:teuthology.orchestra.run.smithi005.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2021-03-09T08:56:55.299 INFO:teuthology.orchestra.run.smithi005.stderr: 2021-03-09T08:56:55.299 INFO:teuthology.orchestra.run.smithi005.stderr:Please consider enabling telemetry to help improve Ceph: 2021-03-09T08:56:55.299 INFO:teuthology.orchestra.run.smithi005.stderr: 2021-03-09T08:56:55.300 INFO:teuthology.orchestra.run.smithi005.stderr: ceph telemetry on 2021-03-09T08:56:55.300 INFO:teuthology.orchestra.run.smithi005.stderr: 2021-03-09T08:56:55.300 INFO:teuthology.orchestra.run.smithi005.stderr:For more information see: 2021-03-09T08:56:55.300 INFO:teuthology.orchestra.run.smithi005.stderr: 2021-03-09T08:56:55.300 INFO:teuthology.orchestra.run.smithi005.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2021-03-09T08:56:55.301 INFO:teuthology.orchestra.run.smithi005.stderr: 2021-03-09T08:56:55.301 INFO:teuthology.orchestra.run.smithi005.stderr:Bootstrap complete. 2021-03-09T08:56:55.301 INFO:teuthology.orchestra.run.smithi005.stderr:Releasing lock 139653478086016 on /run/cephadm/252abbe0-80b5-11eb-907b-001a4aab830c.lock 2021-03-09T08:56:55.302 INFO:teuthology.orchestra.run.smithi005.stderr:Lock 139653478086016 released on /run/cephadm/252abbe0-80b5-11eb-907b-001a4aab830c.lock 2021-03-09T08:56:55.327 INFO:tasks.cephadm:Fetching config... 2021-03-09T08:56:55.329 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2021-03-09T08:56:55.330 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2021-03-09T08:56:55.335 INFO:tasks.cephadm:Fetching client.admin keyring... 2021-03-09T08:56:55.336 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2021-03-09T08:56:55.336 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2021-03-09T08:56:55.383 INFO:tasks.cephadm:Fetching mon keyring... 2021-03-09T08:56:55.384 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2021-03-09T08:56:55.384 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/252abbe0-80b5-11eb-907b-001a4aab830c/mon.a/keyring of=/dev/stdout 2021-03-09T08:56:55.397 INFO:tasks.cephadm:Fetching pub ssh key... 2021-03-09T08:56:55.398 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2021-03-09T08:56:55.398 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2021-03-09T08:56:55.405 INFO:tasks.cephadm:Installing pub ssh key for root users... 2021-03-09T08:56:55.405 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQ59cFYEB+n2raXUNxQMtzJ1oRz3Z8Xit3FDgyZfGXT4n3oRJRGrsp7HH9szPwTTjOZMmPe2WmK8mGa+KAxbIF+Knl6tgTFZrvqeSS7ZZ0nrhVwwi42jmoxPZ7qoK2dECiztxJ/CpkIR82ZLl8VktDT7pRtIhnNgQ6yoTWYH+7a55M2IDsCzcPWwwHoSpteo+g3o4NIJyRX4IZBNuTGqM3995dpR6TAr40FGL5zYNEPXUPcE9mXquVxvSeEFgDMsWLoVMKamEV4WoYdUTZFzFrXzPKN4Ddz+/GliHWFkIIkx9GG4Nf6eISQssiHF+1hJEUt5ao6rIL5dWi4NfC+NmVbWXspGP/KfQJS71Y1sFHE9C2WVI6fs4Drv07Y7qb+0EspKq3o3sFt/uBf3FCIFfExGEWk3qDtT/FGqOkpwA85m81ws0e6aUG8rG1+Lfvo0ETlP+BpQekr6T0SVleo1Qvq9iaLXnuu6rWxmclg/Yftm7mjELtNzTmrABW3hxFzk0= ceph-252abbe0-80b5-11eb-907b-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-03-09T08:56:55.474 INFO:teuthology.orchestra.run.smithi005.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQ59cFYEB+n2raXUNxQMtzJ1oRz3Z8Xit3FDgyZfGXT4n3oRJRGrsp7HH9szPwTTjOZMmPe2WmK8mGa+KAxbIF+Knl6tgTFZrvqeSS7ZZ0nrhVwwi42jmoxPZ7qoK2dECiztxJ/CpkIR82ZLl8VktDT7pRtIhnNgQ6yoTWYH+7a55M2IDsCzcPWwwHoSpteo+g3o4NIJyRX4IZBNuTGqM3995dpR6TAr40FGL5zYNEPXUPcE9mXquVxvSeEFgDMsWLoVMKamEV4WoYdUTZFzFrXzPKN4Ddz+/GliHWFkIIkx9GG4Nf6eISQssiHF+1hJEUt5ao6rIL5dWi4NfC+NmVbWXspGP/KfQJS71Y1sFHE9C2WVI6fs4Drv07Y7qb+0EspKq3o3sFt/uBf3FCIFfExGEWk3qDtT/FGqOkpwA85m81ws0e6aUG8rG1+Lfvo0ETlP+BpQekr6T0SVleo1Qvq9iaLXnuu6rWxmclg/Yftm7mjELtNzTmrABW3hxFzk0= ceph-252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:56:55.486 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQ59cFYEB+n2raXUNxQMtzJ1oRz3Z8Xit3FDgyZfGXT4n3oRJRGrsp7HH9szPwTTjOZMmPe2WmK8mGa+KAxbIF+Knl6tgTFZrvqeSS7ZZ0nrhVwwi42jmoxPZ7qoK2dECiztxJ/CpkIR82ZLl8VktDT7pRtIhnNgQ6yoTWYH+7a55M2IDsCzcPWwwHoSpteo+g3o4NIJyRX4IZBNuTGqM3995dpR6TAr40FGL5zYNEPXUPcE9mXquVxvSeEFgDMsWLoVMKamEV4WoYdUTZFzFrXzPKN4Ddz+/GliHWFkIIkx9GG4Nf6eISQssiHF+1hJEUt5ao6rIL5dWi4NfC+NmVbWXspGP/KfQJS71Y1sFHE9C2WVI6fs4Drv07Y7qb+0EspKq3o3sFt/uBf3FCIFfExGEWk3qDtT/FGqOkpwA85m81ws0e6aUG8rG1+Lfvo0ETlP+BpQekr6T0SVleo1Qvq9iaLXnuu6rWxmclg/Yftm7mjELtNzTmrABW3hxFzk0= ceph-252abbe0-80b5-11eb-907b-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-03-09T08:56:55.515 INFO:teuthology.orchestra.run.smithi146.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQ59cFYEB+n2raXUNxQMtzJ1oRz3Z8Xit3FDgyZfGXT4n3oRJRGrsp7HH9szPwTTjOZMmPe2WmK8mGa+KAxbIF+Knl6tgTFZrvqeSS7ZZ0nrhVwwi42jmoxPZ7qoK2dECiztxJ/CpkIR82ZLl8VktDT7pRtIhnNgQ6yoTWYH+7a55M2IDsCzcPWwwHoSpteo+g3o4NIJyRX4IZBNuTGqM3995dpR6TAr40FGL5zYNEPXUPcE9mXquVxvSeEFgDMsWLoVMKamEV4WoYdUTZFzFrXzPKN4Ddz+/GliHWFkIIkx9GG4Nf6eISQssiHF+1hJEUt5ao6rIL5dWi4NfC+NmVbWXspGP/KfQJS71Y1sFHE9C2WVI6fs4Drv07Y7qb+0EspKq3o3sFt/uBf3FCIFfExGEWk3qDtT/FGqOkpwA85m81ws0e6aUG8rG1+Lfvo0ETlP+BpQekr6T0SVleo1Qvq9iaLXnuu6rWxmclg/Yftm7mjELtNzTmrABW3hxFzk0= ceph-252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:56:55.523 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2021-03-09T08:56:59.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.155+0000 7f156f986700 1 -- 172.21.15.5:0/1450228271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15680f4be0 msgr2=0x7f15680f5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:59.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.155+0000 7f156f986700 1 --2- 172.21.15.5:0/1450228271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15680f4be0 0x7f15680f5000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f1564009cc0 tx=0x7f1564004d10).stop 2021-03-09T08:56:59.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.155+0000 7f156f986700 1 -- 172.21.15.5:0/1450228271 shutdown_connections 2021-03-09T08:56:59.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.155+0000 7f156f986700 1 --2- 172.21.15.5:0/1450228271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15680f4be0 0x7f15680f5000 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:59.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.155+0000 7f156f986700 1 -- 172.21.15.5:0/1450228271 >> 172.21.15.5:0/1450228271 conn(0x7f15680f0340 msgr2=0x7f15680f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:59.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.155+0000 7f156f986700 1 -- 172.21.15.5:0/1450228271 shutdown_connections 2021-03-09T08:56:59.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.155+0000 7f156f986700 1 -- 172.21.15.5:0/1450228271 wait complete. 2021-03-09T08:56:59.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.155+0000 7f156f986700 1 Processor -- start 2021-03-09T08:56:59.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.155+0000 7f156f986700 1 -- start start 2021-03-09T08:56:59.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.155+0000 7f156f986700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15680f4be0 0x7f1568103f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:59.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.155+0000 7f156f986700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1568104420 con 0x7f15680f4be0 2021-03-09T08:56:59.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.155+0000 7f156d722700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15680f4be0 0x7f1568103f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:59.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.155+0000 7f156d722700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15680f4be0 0x7f1568103f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46494/0 (socket says 172.21.15.5:46494) 2021-03-09T08:56:59.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.155+0000 7f156d722700 1 -- 172.21.15.5:0/2501502914 learned_addr learned my addr 172.21.15.5:0/2501502914 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:56:59.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.159+0000 7f156d722700 1 -- 172.21.15.5:0/2501502914 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1568104560 con 0x7f15680f4be0 2021-03-09T08:56:59.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.159+0000 7f156d722700 1 --2- 172.21.15.5:0/2501502914 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15680f4be0 0x7f1568103f30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f156400b740 tx=0x7f1564004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:59.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.159+0000 7f155dffb700 1 -- 172.21.15.5:0/2501502914 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f156400e070 con 0x7f15680f4be0 2021-03-09T08:56:59.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.159+0000 7f156f986700 1 -- 172.21.15.5:0/2501502914 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15681047b0 con 0x7f15680f4be0 2021-03-09T08:56:59.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.159+0000 7f155dffb700 1 -- 172.21.15.5:0/2501502914 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f1564007aa0 con 0x7f15680f4be0 2021-03-09T08:56:59.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.159+0000 7f155dffb700 1 -- 172.21.15.5:0/2501502914 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1564018070 con 0x7f15680f4be0 2021-03-09T08:56:59.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.159+0000 7f156f986700 1 -- 172.21.15.5:0/2501502914 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1568105310 con 0x7f15680f4be0 2021-03-09T08:56:59.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.159+0000 7f155dffb700 1 -- 172.21.15.5:0/2501502914 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 13) v1 ==== 36113+0+0 (secure 0 0 0) 0x7f156401cc90 con 0x7f15680f4be0 2021-03-09T08:56:59.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.159+0000 7f155dffb700 1 --2- 172.21.15.5:0/2501502914 >> v2:172.21.15.5:6800/4269212764 conn(0x7f155402ff00 0x7f15540323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:59.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.159+0000 7f155dffb700 1 -- 172.21.15.5:0/2501502914 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f15640418a0 con 0x7f15680f4be0 2021-03-09T08:56:59.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.159+0000 7f156cf21700 1 --2- 172.21.15.5:0/2501502914 >> v2:172.21.15.5:6800/4269212764 conn(0x7f155402ff00 0x7f15540323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:59.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.159+0000 7f156f986700 1 -- 172.21.15.5:0/2501502914 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15680589f0 con 0x7f15680f4be0 2021-03-09T08:56:59.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.159+0000 7f156cf21700 1 --2- 172.21.15.5:0/2501502914 >> v2:172.21.15.5:6800/4269212764 conn(0x7f155402ff00 0x7f15540323c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f15580096a0 tx=0x7f1558006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:59.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.167+0000 7f155dffb700 1 -- 172.21.15.5:0/2501502914 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f15640253a0 con 0x7f15680f4be0 2021-03-09T08:56:59.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.339+0000 7f156f986700 1 -- 172.21.15.5:0/2501502914 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f15680f9dd0 con 0x7f15680f4be0 2021-03-09T08:56:59.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.347+0000 7f155dffb700 1 -- 172.21.15.5:0/2501502914 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f156403f020 con 0x7f15680f4be0 2021-03-09T08:56:59.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.351+0000 7f156f986700 1 -- 172.21.15.5:0/2501502914 >> v2:172.21.15.5:6800/4269212764 conn(0x7f155402ff00 msgr2=0x7f15540323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:59.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.351+0000 7f156f986700 1 --2- 172.21.15.5:0/2501502914 >> v2:172.21.15.5:6800/4269212764 conn(0x7f155402ff00 0x7f15540323c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f15580096a0 tx=0x7f1558006b40).stop 2021-03-09T08:56:59.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.351+0000 7f156f986700 1 -- 172.21.15.5:0/2501502914 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15680f4be0 msgr2=0x7f1568103f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:56:59.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.351+0000 7f156f986700 1 --2- 172.21.15.5:0/2501502914 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15680f4be0 0x7f1568103f30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f156400b740 tx=0x7f1564004d10).stop 2021-03-09T08:56:59.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.351+0000 7f156f986700 1 -- 172.21.15.5:0/2501502914 shutdown_connections 2021-03-09T08:56:59.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.351+0000 7f156f986700 1 --2- 172.21.15.5:0/2501502914 >> v2:172.21.15.5:6800/4269212764 conn(0x7f155402ff00 0x7f15540323c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:59.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.351+0000 7f156f986700 1 --2- 172.21.15.5:0/2501502914 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15680f4be0 0x7f1568103f30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:59.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.351+0000 7f156f986700 1 -- 172.21.15.5:0/2501502914 >> 172.21.15.5:0/2501502914 conn(0x7f15680f0340 msgr2=0x7f15680f0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:56:59.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.351+0000 7f156f986700 1 -- 172.21.15.5:0/2501502914 shutdown_connections 2021-03-09T08:56:59.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:56:59.351+0000 7f156f986700 1 -- 172.21.15.5:0/2501502914 wait complete. 2021-03-09T08:57:00.300 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi146 2021-03-09T08:57:00.308 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2021-03-09T08:57:00.309 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.conf 2021-03-09T08:57:00.317 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2021-03-09T08:57:00.319 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.client.admin.keyring 2021-03-09T08:57:00.326 INFO:tasks.cephadm:Adding host smithi146 to orchestrator... 2021-03-09T08:57:00.327 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph orch host add smithi146 2021-03-09T08:57:00.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:00 smithi005 bash[12738]: audit 2021-03-09T08:56:59.348130+0000 mon.a (mon.0) 87 : audit [INF] from='client.? 172.21.15.5:0/2501502914' entity='client.admin' 2021-03-09T08:57:05.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:05 smithi005 bash[12738]: audit 2021-03-09T08:57:04.127008+0000 mon.a (mon.0) 88 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:57:05.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:05 smithi005 bash[12738]: cephadm 2021-03-09T08:57:04.127623+0000 mgr.y (mgr.14144) 5 : cephadm [INF] Refreshing smithi005 facts 2021-03-09T08:57:05.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:05 smithi005 bash[12738]: audit 2021-03-09T08:57:04.586620+0000 mon.a (mon.0) 89 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:57:05.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:05 smithi005 bash[12738]: audit 2021-03-09T08:57:04.588350+0000 mon.a (mon.0) 90 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:57:05.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:05 smithi005 bash[12738]: audit 2021-03-09T08:57:04.593996+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:57:06.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:06 smithi005 bash[12738]: cluster 2021-03-09T08:57:05.277246+0000 mgr.y (mgr.14144) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:09.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:08 smithi005 bash[12738]: cluster 2021-03-09T08:57:07.277687+0000 mgr.y (mgr.14144) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:11.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:10 smithi005 bash[12738]: cluster 2021-03-09T08:57:09.278189+0000 mgr.y (mgr.14144) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:13.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:12 smithi005 bash[12738]: cluster 2021-03-09T08:57:11.278586+0000 mgr.y (mgr.14144) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:15.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:14 smithi005 bash[12738]: cluster 2021-03-09T08:57:13.279003+0000 mgr.y (mgr.14144) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:17.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:16 smithi005 bash[12738]: cluster 2021-03-09T08:57:15.279413+0000 mgr.y (mgr.14144) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:19.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:18 smithi005 bash[12738]: cluster 2021-03-09T08:57:17.279800+0000 mgr.y (mgr.14144) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:21.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:20 smithi005 bash[12738]: cluster 2021-03-09T08:57:19.280162+0000 mgr.y (mgr.14144) 13 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:23.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:22 smithi005 bash[12738]: cluster 2021-03-09T08:57:21.280625+0000 mgr.y (mgr.14144) 14 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:25.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:24 smithi005 bash[12738]: cluster 2021-03-09T08:57:23.281013+0000 mgr.y (mgr.14144) 15 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:26.886 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffb59e700 1 -- 172.21.15.146:0/2646751354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faffc0f7d30 msgr2=0x7faffc0f8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:57:26.886 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffb59e700 1 --2- 172.21.15.146:0/2646751354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faffc0f7d30 0x7faffc0f8150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7faff4004d00 tx=0x7faff4009b30).stop 2021-03-09T08:57:26.887 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffb59e700 1 -- 172.21.15.146:0/2646751354 shutdown_connections 2021-03-09T08:57:26.887 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffb59e700 1 --2- 172.21.15.146:0/2646751354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faffc0f7d30 0x7faffc0f8150 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:26.887 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffb59e700 1 -- 172.21.15.146:0/2646751354 >> 172.21.15.146:0/2646751354 conn(0x7faffc060930 msgr2=0x7faffc0f59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:57:26.887 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffb59e700 1 -- 172.21.15.146:0/2646751354 shutdown_connections 2021-03-09T08:57:26.888 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffb59e700 1 -- 172.21.15.146:0/2646751354 wait complete. 2021-03-09T08:57:26.889 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffb59e700 1 Processor -- start 2021-03-09T08:57:26.889 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffb59e700 1 -- start start 2021-03-09T08:57:26.890 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffb59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faffc0f7d30 0x7faffc107080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:26.890 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffb59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc107570 con 0x7faffc0f7d30 2021-03-09T08:57:26.890 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffa59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faffc0f7d30 0x7faffc107080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:26.890 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffa59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faffc0f7d30 0x7faffc107080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.146:50962/0 (socket says 172.21.15.146:50962) 2021-03-09T08:57:26.890 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffa59c700 1 -- 172.21.15.146:0/735501307 learned_addr learned my addr 172.21.15.146:0/735501307 (peer_addr_for_me v2:172.21.15.146:0/0) 2021-03-09T08:57:26.891 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffa59c700 1 -- 172.21.15.146:0/735501307 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc1076b0 con 0x7faffc0f7d30 2021-03-09T08:57:26.891 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffa59c700 1 --2- 172.21.15.146:0/735501307 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faffc0f7d30 0x7faffc107080 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7faff4007ed0 tx=0x7faff4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:26.891 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faff2ffd700 1 -- 172.21.15.146:0/735501307 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faff400e030 con 0x7faffc0f7d30 2021-03-09T08:57:26.891 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffb59e700 1 -- 172.21.15.146:0/735501307 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faffc107900 con 0x7faffc0f7d30 2021-03-09T08:57:26.892 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faff2ffd700 1 -- 172.21.15.146:0/735501307 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7faff4013aa0 con 0x7faffc0f7d30 2021-03-09T08:57:26.892 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faff2ffd700 1 -- 172.21.15.146:0/735501307 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faff4018070 con 0x7faffc0f7d30 2021-03-09T08:57:26.892 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.882+0000 7faffb59e700 1 -- 172.21.15.146:0/735501307 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faffc108460 con 0x7faffc0f7d30 2021-03-09T08:57:26.893 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.886+0000 7faff2ffd700 1 -- 172.21.15.146:0/735501307 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 13) v1 ==== 36113+0+0 (secure 0 0 0) 0x7faff401d990 con 0x7faffc0f7d30 2021-03-09T08:57:26.893 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.886+0000 7faffb59e700 1 -- 172.21.15.146:0/735501307 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faffc0589f0 con 0x7faffc0f7d30 2021-03-09T08:57:26.894 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.886+0000 7faff2ffd700 1 --2- 172.21.15.146:0/735501307 >> v2:172.21.15.5:6800/4269212764 conn(0x7fafe802feb0 0x7fafe8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:26.894 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.886+0000 7faff2ffd700 1 -- 172.21.15.146:0/735501307 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7faff40427c0 con 0x7faffc0f7d30 2021-03-09T08:57:26.895 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.886+0000 7faff9d9b700 1 --2- 172.21.15.146:0/735501307 >> v2:172.21.15.5:6800/4269212764 conn(0x7fafe802feb0 0x7fafe8032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:26.895 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.886+0000 7faff9d9b700 1 --2- 172.21.15.146:0/735501307 >> v2:172.21.15.5:6800/4269212764 conn(0x7fafe802feb0 0x7fafe8032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fafe40089e0 tx=0x7fafe4006ec0).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:26.900 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:26.894+0000 7faff2ffd700 1 -- 172.21.15.146:0/735501307 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7faff401b370 con 0x7faffc0f7d30 2021-03-09T08:57:27.090 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:27.086+0000 7faffb59e700 1 -- 172.21.15.146:0/735501307 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi146", "target": ["mon-mgr", ""]}) v1 -- 0x7faffc1086a0 con 0x7fafe802feb0 2021-03-09T08:57:27.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:26 smithi005 bash[12738]: cluster 2021-03-09T08:57:25.281414+0000 mgr.y (mgr.14144) 16 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:27.618 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 09 08:57:27 smithi005 bash[12977]: Warning: Permanently added 'smithi146,172.21.15.146' (ECDSA) to the list of known hosts. 2021-03-09T08:57:27.885 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:27.878+0000 7faff2ffd700 1 -- 172.21.15.146:0/735501307 <== mgr.14144 v2:172.21.15.5:6800/4269212764 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7faffc1086a0 con 0x7fafe802feb0 2021-03-09T08:57:27.886 INFO:teuthology.orchestra.run.smithi146.stdout:Added host 'smithi146' 2021-03-09T08:57:27.888 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:27.882+0000 7faffb59e700 1 -- 172.21.15.146:0/735501307 >> v2:172.21.15.5:6800/4269212764 conn(0x7fafe802feb0 msgr2=0x7fafe8032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:57:27.888 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:27.882+0000 7faffb59e700 1 --2- 172.21.15.146:0/735501307 >> v2:172.21.15.5:6800/4269212764 conn(0x7fafe802feb0 0x7fafe8032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fafe40089e0 tx=0x7fafe4006ec0).stop 2021-03-09T08:57:27.889 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:27.882+0000 7faffb59e700 1 -- 172.21.15.146:0/735501307 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faffc0f7d30 msgr2=0x7faffc107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:57:27.889 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:27.882+0000 7faffb59e700 1 --2- 172.21.15.146:0/735501307 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faffc0f7d30 0x7faffc107080 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7faff4007ed0 tx=0x7faff4009b30).stop 2021-03-09T08:57:27.889 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:27.882+0000 7faffb59e700 1 -- 172.21.15.146:0/735501307 shutdown_connections 2021-03-09T08:57:27.889 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:27.882+0000 7faffb59e700 1 --2- 172.21.15.146:0/735501307 >> v2:172.21.15.5:6800/4269212764 conn(0x7fafe802feb0 0x7fafe8032370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:27.890 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:27.882+0000 7faffb59e700 1 --2- 172.21.15.146:0/735501307 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faffc0f7d30 0x7faffc107080 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:27.890 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:27.882+0000 7faffb59e700 1 -- 172.21.15.146:0/735501307 >> 172.21.15.146:0/735501307 conn(0x7faffc060930 msgr2=0x7faffc0f6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:57:27.890 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:27.882+0000 7faffb59e700 1 -- 172.21.15.146:0/735501307 shutdown_connections 2021-03-09T08:57:27.890 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:27.882+0000 7faffb59e700 1 -- 172.21.15.146:0/735501307 wait complete. 2021-03-09T08:57:28.626 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph orch host ls --format=json 2021-03-09T08:57:29.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:28 smithi005 bash[12738]: audit 2021-03-09T08:57:27.091695+0000 mgr.y (mgr.14144) 17 : audit [DBG] from='client.14162 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi146", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:57:29.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:28 smithi005 bash[12738]: cluster 2021-03-09T08:57:27.282106+0000 mgr.y (mgr.14144) 18 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:29.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:28 smithi005 bash[12738]: audit 2021-03-09T08:57:27.881973+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:57:29.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:28 smithi005 bash[12738]: cephadm 2021-03-09T08:57:27.882934+0000 mgr.y (mgr.14144) 19 : cephadm [INF] Added host smithi146 2021-03-09T08:57:29.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:28 smithi005 bash[12738]: audit 2021-03-09T08:57:27.886238+0000 mon.a (mon.0) 93 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:57:29.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:28 smithi005 bash[12738]: audit 2021-03-09T08:57:28.321669+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:57:30.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:30 smithi005 bash[12738]: audit 2021-03-09T08:57:29.246250+0000 mon.a (mon.0) 95 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:57:30.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:30 smithi005 bash[12738]: cluster 2021-03-09T08:57:29.282680+0000 mgr.y (mgr.14144) 20 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:32.741 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.734+0000 7f8a4891e700 1 -- 172.21.15.146:0/4237648338 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4405fb70 msgr2=0x7f8a440f64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:57:32.742 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.734+0000 7f8a4891e700 1 --2- 172.21.15.146:0/4237648338 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4405fb70 0x7f8a440f64e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f8a34004d00 tx=0x7f8a34009b30).stop 2021-03-09T08:57:32.742 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.734+0000 7f8a4891e700 1 -- 172.21.15.146:0/4237648338 shutdown_connections 2021-03-09T08:57:32.742 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.734+0000 7f8a4891e700 1 --2- 172.21.15.146:0/4237648338 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4405fb70 0x7f8a440f64e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.734+0000 7f8a4891e700 1 -- 172.21.15.146:0/4237648338 >> 172.21.15.146:0/4237648338 conn(0x7f8a440f0170 msgr2=0x7f8a440f2590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:57:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.734+0000 7f8a4891e700 1 -- 172.21.15.146:0/4237648338 shutdown_connections 2021-03-09T08:57:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.734+0000 7f8a4891e700 1 -- 172.21.15.146:0/4237648338 wait complete. 2021-03-09T08:57:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.738+0000 7f8a4891e700 1 Processor -- start 2021-03-09T08:57:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.738+0000 7f8a4891e700 1 -- start start 2021-03-09T08:57:32.745 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.738+0000 7f8a4891e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4405fb70 0x7f8a4410b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:32.745 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.738+0000 7f8a4891e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4410b950 con 0x7f8a4405fb70 2021-03-09T08:57:32.745 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.738+0000 7f8a42d9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4405fb70 0x7f8a4410b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:32.745 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.738+0000 7f8a42d9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4405fb70 0x7f8a4410b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.146:50968/0 (socket says 172.21.15.146:50968) 2021-03-09T08:57:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.738+0000 7f8a42d9d700 1 -- 172.21.15.146:0/1061930269 learned_addr learned my addr 172.21.15.146:0/1061930269 (peer_addr_for_me v2:172.21.15.146:0/0) 2021-03-09T08:57:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.738+0000 7f8a42d9d700 1 -- 172.21.15.146:0/1061930269 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4410ba90 con 0x7f8a4405fb70 2021-03-09T08:57:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.738+0000 7f8a42d9d700 1 --2- 172.21.15.146:0/1061930269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4405fb70 0x7f8a4410b460 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f8a34007ed0 tx=0x7f8a34009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.738+0000 7f8a3bfff700 1 -- 172.21.15.146:0/1061930269 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a3400e030 con 0x7f8a4405fb70 2021-03-09T08:57:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.738+0000 7f8a4891e700 1 -- 172.21.15.146:0/1061930269 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4410bce0 con 0x7f8a4405fb70 2021-03-09T08:57:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.738+0000 7f8a4891e700 1 -- 172.21.15.146:0/1061930269 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4410c840 con 0x7f8a4405fb70 2021-03-09T08:57:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.738+0000 7f8a3bfff700 1 -- 172.21.15.146:0/1061930269 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8a34013aa0 con 0x7f8a4405fb70 2021-03-09T08:57:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.738+0000 7f8a3bfff700 1 -- 172.21.15.146:0/1061930269 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a34018070 con 0x7f8a4405fb70 2021-03-09T08:57:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.738+0000 7f8a4891e700 1 -- 172.21.15.146:0/1061930269 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a440589f0 con 0x7f8a4405fb70 2021-03-09T08:57:32.748 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.742+0000 7f8a3bfff700 1 -- 172.21.15.146:0/1061930269 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 13) v1 ==== 36113+0+0 (secure 0 0 0) 0x7f8a34013c10 con 0x7f8a4405fb70 2021-03-09T08:57:32.749 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.742+0000 7f8a3bfff700 1 --2- 172.21.15.146:0/1061930269 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8a3002ff00 0x7f8a300323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:32.749 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.742+0000 7f8a3bfff700 1 -- 172.21.15.146:0/1061930269 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f8a340419f0 con 0x7f8a4405fb70 2021-03-09T08:57:32.749 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.742+0000 7f8a3a59c700 1 --2- 172.21.15.146:0/1061930269 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8a3002ff00 0x7f8a300323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:32.750 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.742+0000 7f8a3a59c700 1 --2- 172.21.15.146:0/1061930269 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8a3002ff00 0x7f8a300323c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f8a2c0096a0 tx=0x7f8a2c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:32.758 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.754+0000 7f8a3bfff700 1 -- 172.21.15.146:0/1061930269 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f8a34025b80 con 0x7f8a4405fb70 2021-03-09T08:57:32.953 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.946+0000 7f8a4891e700 1 -- 172.21.15.146:0/1061930269 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a440f8bf0 con 0x7f8a3002ff00 2021-03-09T08:57:32.956 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.950+0000 7f8a3bfff700 1 -- 172.21.15.146:0/1061930269 <== mgr.14144 v2:172.21.15.5:6800/4269212764 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+153 (secure 0 0 0) 0x7f8a440f8bf0 con 0x7f8a3002ff00 2021-03-09T08:57:32.956 INFO:teuthology.orchestra.run.smithi146.stdout: 2021-03-09T08:57:32.957 INFO:teuthology.orchestra.run.smithi146.stdout:[{"addr": "smithi005", "hostname": "smithi005", "labels": [], "status": ""}, {"addr": "smithi146", "hostname": "smithi146", "labels": [], "status": ""}] 2021-03-09T08:57:32.959 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.954+0000 7f8a4891e700 1 -- 172.21.15.146:0/1061930269 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8a3002ff00 msgr2=0x7f8a300323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:57:32.960 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.954+0000 7f8a4891e700 1 --2- 172.21.15.146:0/1061930269 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8a3002ff00 0x7f8a300323c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f8a2c0096a0 tx=0x7f8a2c006b40).stop 2021-03-09T08:57:32.960 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.954+0000 7f8a4891e700 1 -- 172.21.15.146:0/1061930269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4405fb70 msgr2=0x7f8a4410b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:57:32.960 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.954+0000 7f8a4891e700 1 --2- 172.21.15.146:0/1061930269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4405fb70 0x7f8a4410b460 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f8a34007ed0 tx=0x7f8a34009b30).stop 2021-03-09T08:57:32.960 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.954+0000 7f8a4891e700 1 -- 172.21.15.146:0/1061930269 shutdown_connections 2021-03-09T08:57:32.961 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.954+0000 7f8a4891e700 1 --2- 172.21.15.146:0/1061930269 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8a3002ff00 0x7f8a300323c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:32.961 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.954+0000 7f8a4891e700 1 --2- 172.21.15.146:0/1061930269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a4405fb70 0x7f8a4410b460 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:32.961 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.954+0000 7f8a4891e700 1 -- 172.21.15.146:0/1061930269 >> 172.21.15.146:0/1061930269 conn(0x7f8a440f0170 msgr2=0x7f8a440f5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:57:32.961 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.954+0000 7f8a4891e700 1 -- 172.21.15.146:0/1061930269 shutdown_connections 2021-03-09T08:57:32.962 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:32.954+0000 7f8a4891e700 1 -- 172.21.15.146:0/1061930269 wait complete. 2021-03-09T08:57:33.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:32 smithi005 bash[12738]: cluster 2021-03-09T08:57:31.283190+0000 mgr.y (mgr.14144) 21 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:33.870 INFO:tasks.cephadm:Setting crush tunables to default 2021-03-09T08:57:33.870 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph osd crush tunables default 2021-03-09T08:57:34.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:33 smithi005 bash[12738]: audit 2021-03-09T08:57:32.954217+0000 mgr.y (mgr.14144) 22 : audit [DBG] from='client.14164 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-09T08:57:35.194 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:34 smithi005 bash[12738]: cluster 2021-03-09T08:57:33.283713+0000 mgr.y (mgr.14144) 23 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:36.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb9a8a9700 1 -- 172.21.15.5:0/3809105964 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb940f7d30 msgr2=0x7fbb940f8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:57:36.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb9a8a9700 1 --2- 172.21.15.5:0/3809105964 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb940f7d30 0x7fbb940f8150 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbb84004d00 tx=0x7fbb84009b30).stop 2021-03-09T08:57:36.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb9a8a9700 1 -- 172.21.15.5:0/3809105964 shutdown_connections 2021-03-09T08:57:36.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb9a8a9700 1 --2- 172.21.15.5:0/3809105964 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb940f7d30 0x7fbb940f8150 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:36.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb9a8a9700 1 -- 172.21.15.5:0/3809105964 >> 172.21.15.5:0/3809105964 conn(0x7fbb94060930 msgr2=0x7fbb940f59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:57:36.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb9a8a9700 1 -- 172.21.15.5:0/3809105964 shutdown_connections 2021-03-09T08:57:36.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb9a8a9700 1 -- 172.21.15.5:0/3809105964 wait complete. 2021-03-09T08:57:36.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb9a8a9700 1 Processor -- start 2021-03-09T08:57:36.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb9a8a9700 1 -- start start 2021-03-09T08:57:36.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb9a8a9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb940f7d30 0x7fbb941070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:36.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb9a8a9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb941075b0 con 0x7fbb940f7d30 2021-03-09T08:57:36.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb998a7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb940f7d30 0x7fbb941070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:36.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb998a7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb940f7d30 0x7fbb941070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46502/0 (socket says 172.21.15.5:46502) 2021-03-09T08:57:36.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb998a7700 1 -- 172.21.15.5:0/2551983255 learned_addr learned my addr 172.21.15.5:0/2551983255 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:57:36.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb998a7700 1 -- 172.21.15.5:0/2551983255 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb941076f0 con 0x7fbb940f7d30 2021-03-09T08:57:36.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb998a7700 1 --2- 172.21.15.5:0/2551983255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb940f7d30 0x7fbb941070c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fbb84004f50 tx=0x7fbb84009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:36.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb8a7fc700 1 -- 172.21.15.5:0/2551983255 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb8400e030 con 0x7fbb940f7d30 2021-03-09T08:57:36.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb8a7fc700 1 -- 172.21.15.5:0/2551983255 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7fbb84009430 con 0x7fbb940f7d30 2021-03-09T08:57:36.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb8a7fc700 1 -- 172.21.15.5:0/2551983255 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb84018070 con 0x7fbb940f7d30 2021-03-09T08:57:36.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb9a8a9700 1 -- 172.21.15.5:0/2551983255 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb94107940 con 0x7fbb940f7d30 2021-03-09T08:57:36.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.971+0000 7fbb9a8a9700 1 -- 172.21.15.5:0/2551983255 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb941084a0 con 0x7fbb940f7d30 2021-03-09T08:57:36.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.975+0000 7fbb8a7fc700 1 -- 172.21.15.5:0/2551983255 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 13) v1 ==== 36113+0+0 (secure 0 0 0) 0x7fbb84013620 con 0x7fbb940f7d30 2021-03-09T08:57:36.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.975+0000 7fbb8a7fc700 1 --2- 172.21.15.5:0/2551983255 >> v2:172.21.15.5:6800/4269212764 conn(0x7fbb8002feb0 0x7fbb80032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:36.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.975+0000 7fbb8a7fc700 1 -- 172.21.15.5:0/2551983255 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7fbb840418c0 con 0x7fbb940f7d30 2021-03-09T08:57:36.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.975+0000 7fbb990a6700 1 --2- 172.21.15.5:0/2551983255 >> v2:172.21.15.5:6800/4269212764 conn(0x7fbb8002feb0 0x7fbb80032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:36.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.975+0000 7fbb9a8a9700 1 -- 172.21.15.5:0/2551983255 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb940589f0 con 0x7fbb940f7d30 2021-03-09T08:57:36.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.975+0000 7fbb990a6700 1 --2- 172.21.15.5:0/2551983255 >> v2:172.21.15.5:6800/4269212764 conn(0x7fbb8002feb0 0x7fbb80032370 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fbb900089e0 tx=0x7fbb90006ec0).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:36.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:36.983+0000 7fbb8a7fc700 1 -- 172.21.15.5:0/2551983255 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7fbb84013de0 con 0x7fbb940f7d30 2021-03-09T08:57:37.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:36 smithi005 bash[12738]: cluster 2021-03-09T08:57:35.284216+0000 mgr.y (mgr.14144) 24 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:37.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:36 smithi005 bash[12738]: audit 2021-03-09T08:57:36.295548+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:57:37.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:36 smithi005 bash[12738]: audit 2021-03-09T08:57:36.743470+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:57:37.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:36 smithi005 bash[12738]: audit 2021-03-09T08:57:36.748803+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:57:37.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:37.155+0000 7fbb9a8a9700 1 -- 172.21.15.5:0/2551983255 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fbb940fcff0 con 0x7fbb940f7d30 2021-03-09T08:57:37.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:37.879+0000 7fbb8a7fc700 1 -- 172.21.15.5:0/2551983255 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fbb84025b40 con 0x7fbb940f7d30 2021-03-09T08:57:37.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:37.883+0000 7fbb9a8a9700 1 -- 172.21.15.5:0/2551983255 >> v2:172.21.15.5:6800/4269212764 conn(0x7fbb8002feb0 msgr2=0x7fbb80032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:57:37.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:37.883+0000 7fbb9a8a9700 1 --2- 172.21.15.5:0/2551983255 >> v2:172.21.15.5:6800/4269212764 conn(0x7fbb8002feb0 0x7fbb80032370 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fbb900089e0 tx=0x7fbb90006ec0).stop 2021-03-09T08:57:37.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:37.883+0000 7fbb9a8a9700 1 -- 172.21.15.5:0/2551983255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb940f7d30 msgr2=0x7fbb941070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:57:37.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:37.883+0000 7fbb9a8a9700 1 --2- 172.21.15.5:0/2551983255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb940f7d30 0x7fbb941070c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fbb84004f50 tx=0x7fbb84009b30).stop 2021-03-09T08:57:37.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:37.883+0000 7fbb9a8a9700 1 -- 172.21.15.5:0/2551983255 shutdown_connections 2021-03-09T08:57:37.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:37.883+0000 7fbb9a8a9700 1 --2- 172.21.15.5:0/2551983255 >> v2:172.21.15.5:6800/4269212764 conn(0x7fbb8002feb0 0x7fbb80032370 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:37.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:37.883+0000 7fbb9a8a9700 1 --2- 172.21.15.5:0/2551983255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb940f7d30 0x7fbb941070c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:37.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:37.883+0000 7fbb9a8a9700 1 -- 172.21.15.5:0/2551983255 >> 172.21.15.5:0/2551983255 conn(0x7fbb94060930 msgr2=0x7fbb940f4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:57:37.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:37.883+0000 7fbb9a8a9700 1 -- 172.21.15.5:0/2551983255 shutdown_connections 2021-03-09T08:57:37.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:37.883+0000 7fbb9a8a9700 1 -- 172.21.15.5:0/2551983255 wait complete. 2021-03-09T08:57:37.890 INFO:teuthology.orchestra.run.smithi005.stderr:adjusted tunables profile to default 2021-03-09T08:57:38.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:37 smithi005 bash[12738]: cephadm 2021-03-09T08:57:36.296426+0000 mgr.y (mgr.14144) 25 : cephadm [INF] Refreshing smithi146 facts 2021-03-09T08:57:38.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:37 smithi005 bash[12738]: audit 2021-03-09T08:57:37.159871+0000 mon.a (mon.0) 99 : audit [INF] from='client.? 172.21.15.5:0/2551983255' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2021-03-09T08:57:38.604 INFO:tasks.cephadm:Adding mon.c on smithi005 2021-03-09T08:57:38.605 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph orch daemon add mon 'smithi005:[v2:172.21.15.5:3301,v1:172.21.15.5:6790]=c' 2021-03-09T08:57:39.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:38 smithi005 bash[12738]: cluster 2021-03-09T08:57:37.284650+0000 mgr.y (mgr.14144) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:39.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:38 smithi005 bash[12738]: audit 2021-03-09T08:57:37.883199+0000 mon.a (mon.0) 100 : audit [INF] from='client.? 172.21.15.5:0/2551983255' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2021-03-09T08:57:39.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:38 smithi005 bash[12738]: cluster 2021-03-09T08:57:37.883386+0000 mon.a (mon.0) 101 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-09T08:57:41.338 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:40 smithi005 bash[12738]: cluster 2021-03-09T08:57:39.285070+0000 mgr.y (mgr.14144) 27 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:41.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.619+0000 7f467d984700 1 -- 172.21.15.5:0/3422227219 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f467805fb70 msgr2=0x7f46780f8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:57:41.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.619+0000 7f467d984700 1 --2- 172.21.15.5:0/3422227219 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f467805fb70 0x7f46780f8710 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f4674004d00 tx=0x7f4674009b30).stop 2021-03-09T08:57:41.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.619+0000 7f467d984700 1 -- 172.21.15.5:0/3422227219 shutdown_connections 2021-03-09T08:57:41.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.619+0000 7f467d984700 1 --2- 172.21.15.5:0/3422227219 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f467805fb70 0x7f46780f8710 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:41.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.619+0000 7f467d984700 1 -- 172.21.15.5:0/3422227219 >> 172.21.15.5:0/3422227219 conn(0x7f46780f0170 msgr2=0x7f46780f25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:57:41.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.619+0000 7f467d984700 1 -- 172.21.15.5:0/3422227219 shutdown_connections 2021-03-09T08:57:41.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.619+0000 7f467d984700 1 -- 172.21.15.5:0/3422227219 wait complete. 2021-03-09T08:57:41.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.619+0000 7f467d984700 1 Processor -- start 2021-03-09T08:57:41.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.619+0000 7f467d984700 1 -- start start 2021-03-09T08:57:41.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.619+0000 7f467d984700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f467805fb70 0x7f4678107050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:41.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.619+0000 7f467d984700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4678107540 con 0x7f467805fb70 2021-03-09T08:57:41.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.619+0000 7f467c982700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f467805fb70 0x7f4678107050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:41.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.619+0000 7f467c982700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f467805fb70 0x7f4678107050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46508/0 (socket says 172.21.15.5:46508) 2021-03-09T08:57:41.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.619+0000 7f467c982700 1 -- 172.21.15.5:0/269399670 learned_addr learned my addr 172.21.15.5:0/269399670 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:57:41.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.623+0000 7f467c982700 1 -- 172.21.15.5:0/269399670 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4678107680 con 0x7f467805fb70 2021-03-09T08:57:41.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.623+0000 7f467c982700 1 --2- 172.21.15.5:0/269399670 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f467805fb70 0x7f4678107050 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f4674007ed0 tx=0x7f4674009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:41.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.623+0000 7f466d7fa700 1 -- 172.21.15.5:0/269399670 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f467400e030 con 0x7f467805fb70 2021-03-09T08:57:41.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.623+0000 7f466d7fa700 1 -- 172.21.15.5:0/269399670 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f4674013aa0 con 0x7f467805fb70 2021-03-09T08:57:41.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.623+0000 7f466d7fa700 1 -- 172.21.15.5:0/269399670 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4674018070 con 0x7f467805fb70 2021-03-09T08:57:41.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.623+0000 7f467d984700 1 -- 172.21.15.5:0/269399670 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46781078d0 con 0x7f467805fb70 2021-03-09T08:57:41.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.623+0000 7f467d984700 1 -- 172.21.15.5:0/269399670 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4678108430 con 0x7f467805fb70 2021-03-09T08:57:41.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.623+0000 7f466d7fa700 1 -- 172.21.15.5:0/269399670 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 13) v1 ==== 36113+0+0 (secure 0 0 0) 0x7f4674013c10 con 0x7f467805fb70 2021-03-09T08:57:41.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.623+0000 7f466d7fa700 1 --2- 172.21.15.5:0/269399670 >> v2:172.21.15.5:6800/4269212764 conn(0x7f466002ff00 0x7f46600323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:41.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.623+0000 7f466d7fa700 1 -- 172.21.15.5:0/269399670 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f46740419a0 con 0x7f467805fb70 2021-03-09T08:57:41.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.623+0000 7f466ffff700 1 --2- 172.21.15.5:0/269399670 >> v2:172.21.15.5:6800/4269212764 conn(0x7f466002ff00 0x7f46600323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:41.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.623+0000 7f467d984700 1 -- 172.21.15.5:0/269399670 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46780589f0 con 0x7f467805fb70 2021-03-09T08:57:41.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.623+0000 7f466ffff700 1 --2- 172.21.15.5:0/269399670 >> v2:172.21.15.5:6800/4269212764 conn(0x7f466002ff00 0x7f46600323c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f46680096a0 tx=0x7f4668006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:41.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.631+0000 7f466d7fa700 1 -- 172.21.15.5:0/269399670 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f4674013620 con 0x7f467805fb70 2021-03-09T08:57:41.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:41.827+0000 7f467d984700 1 -- 172.21.15.5:0/269399670 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi005:[v2:172.21.15.5:3301,v1:172.21.15.5:6790]=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f46780fada0 con 0x7f466002ff00 2021-03-09T08:57:42.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:41 smithi005 bash[12738]: audit 2021-03-09T08:57:41.834384+0000 mon.a (mon.0) 102 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-03-09T08:57:42.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:41 smithi005 bash[12738]: audit 2021-03-09T08:57:41.836120+0000 mon.a (mon.0) 103 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:57:43.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:42 smithi005 bash[12738]: cluster 2021-03-09T08:57:41.285508+0000 mgr.y (mgr.14144) 28 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:43.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:42 smithi005 bash[12738]: audit 2021-03-09T08:57:41.830454+0000 mgr.y (mgr.14144) 29 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi005:[v2:172.21.15.5:3301,v1:172.21.15.5:6790]=c", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:57:43.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:42 smithi005 bash[12738]: cephadm 2021-03-09T08:57:41.837559+0000 mgr.y (mgr.14144) 30 : cephadm [INF] Deploying daemon mon.c on smithi005 2021-03-09T08:57:45.239 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:44 smithi005 bash[12738]: cluster 2021-03-09T08:57:43.285927+0000 mgr.y (mgr.14144) 31 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:46.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:45 smithi005 bash[12738]: audit 2021-03-09T08:57:45.481182+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-09T08:57:46.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:45 smithi005 bash[12738]: audit 2021-03-09T08:57:45.484212+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-09T08:57:46.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:46.311+0000 7f466d7fa700 1 -- 172.21.15.5:0/269399670 <== mgr.14144 v2:172.21.15.5:6800/4269212764 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+35 (secure 0 0 0) 0x7f46780fada0 con 0x7f466002ff00 2021-03-09T08:57:46.316 INFO:teuthology.orchestra.run.smithi005.stdout:Deployed mon.c on host 'smithi005' 2021-03-09T08:57:46.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:46.315+0000 7f467d984700 1 -- 172.21.15.5:0/269399670 >> v2:172.21.15.5:6800/4269212764 conn(0x7f466002ff00 msgr2=0x7f46600323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:57:46.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:46.315+0000 7f467d984700 1 --2- 172.21.15.5:0/269399670 >> v2:172.21.15.5:6800/4269212764 conn(0x7f466002ff00 0x7f46600323c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f46680096a0 tx=0x7f4668006b40).stop 2021-03-09T08:57:46.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:46.315+0000 7f467d984700 1 -- 172.21.15.5:0/269399670 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f467805fb70 msgr2=0x7f4678107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:57:46.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:46.315+0000 7f467d984700 1 --2- 172.21.15.5:0/269399670 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f467805fb70 0x7f4678107050 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f4674007ed0 tx=0x7f4674009b30).stop 2021-03-09T08:57:46.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:46.315+0000 7f467d984700 1 -- 172.21.15.5:0/269399670 shutdown_connections 2021-03-09T08:57:46.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:46.315+0000 7f467d984700 1 --2- 172.21.15.5:0/269399670 >> v2:172.21.15.5:6800/4269212764 conn(0x7f466002ff00 0x7f46600323c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:46.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:46.315+0000 7f467d984700 1 --2- 172.21.15.5:0/269399670 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f467805fb70 0x7f4678107050 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:46.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:46.315+0000 7f467d984700 1 -- 172.21.15.5:0/269399670 >> 172.21.15.5:0/269399670 conn(0x7f46780f0170 msgr2=0x7f46780f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:57:46.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:46.315+0000 7f467d984700 1 -- 172.21.15.5:0/269399670 shutdown_connections 2021-03-09T08:57:46.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:46.315+0000 7f467d984700 1 -- 172.21.15.5:0/269399670 wait complete. 2021-03-09T08:57:47.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:46 smithi005 bash[12738]: cluster 2021-03-09T08:57:45.286377+0000 mgr.y (mgr.14144) 32 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:47.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:46 smithi005 bash[12738]: audit 2021-03-09T08:57:46.309137+0000 mon.a (mon.0) 106 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:57:47.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:46 smithi005 bash[12738]: audit 2021-03-09T08:57:46.312547+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:57:47.650 DEBUG:teuthology.orchestra.run.smithi005:mon.c> sudo journalctl -f -n 0 -u ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service 2021-03-09T08:57:47.653 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-03-09T08:57:47.654 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph mon dump -f json 2021-03-09T08:57:47.698 INFO:journalctl@ceph.mon.c.smithi005.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-09T08:57:48.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.931+0000 7f202a18b700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-09T08:57:48.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.931+0000 7f202a18b700 0 ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev), process ceph-mon, pid 6 2021-03-09T08:57:48.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.931+0000 7f202a18b700 0 pidfile_write: ignore empty --pid-file 2021-03-09T08:57:48.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 0 load: jerasure load: lrc load: isa 2021-03-09T08:57:48.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: RocksDB version: 6.8.1 2021-03-09T08:57:48.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-09T08:57:48.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Compile date Mar 9 2021 2021-03-09T08:57:48.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: DB SUMMARY 2021-03-09T08:57:48.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: CURRENT file: CURRENT 2021-03-09T08:57:48.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-09T08:57:48.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2021-03-09T08:57:48.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2021-03-09T08:57:48.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000003.log size: 511 ; 2021-03-09T08:57:48.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.error_if_exists: 0 2021-03-09T08:57:48.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.create_if_missing: 0 2021-03-09T08:57:48.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.paranoid_checks: 1 2021-03-09T08:57:48.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.env: 0x55adc8c051c0 2021-03-09T08:57:48.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.fs: Posix File System 2021-03-09T08:57:48.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.info_log: 0x55add256f120 2021-03-09T08:57:48.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-09T08:57:48.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.statistics: (nil) 2021-03-09T08:57:48.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.use_fsync: 0 2021-03-09T08:57:48.374 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_log_file_size: 0 2021-03-09T08:57:48.374 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-09T08:57:48.374 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-09T08:57:48.374 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-09T08:57:48.375 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-09T08:57:48.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.allow_fallocate: 1 2021-03-09T08:57:48.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-09T08:57:48.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-09T08:57:48.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.use_direct_reads: 0 2021-03-09T08:57:48.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-09T08:57:48.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-09T08:57:48.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.db_log_dir: 2021-03-09T08:57:48.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2021-03-09T08:57:48.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-09T08:57:48.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_subcompactions: 1 2021-03-09T08:57:48.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_background_flushes: -1 2021-03-09T08:57:48.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-09T08:57:48.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-09T08:57:48.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-09T08:57:48.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-09T08:57:48.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-09T08:57:48.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-09T08:57:48.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-09T08:57:48.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.write_buffer_manager: 0x55add25c3ce0 2021-03-09T08:57:48.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-09T08:57:48.382 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-09T08:57:48.382 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-09T08:57:48.382 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-09T08:57:48.382 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-09T08:57:48.383 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-09T08:57:48.383 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-09T08:57:48.383 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-09T08:57:48.383 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-09T08:57:48.383 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.unordered_write: 0 2021-03-09T08:57:48.384 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-09T08:57:48.384 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-09T08:57:48.384 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-09T08:57:48.384 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-09T08:57:48.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.row_cache: None 2021-03-09T08:57:48.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.wal_filter: None 2021-03-09T08:57:48.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-09T08:57:48.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-09T08:57:48.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.preserve_deletes: 0 2021-03-09T08:57:48.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.two_write_queues: 0 2021-03-09T08:57:48.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-09T08:57:48.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.atomic_flush: 0 2021-03-09T08:57:48.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-09T08:57:48.387 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-09T08:57:48.387 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-09T08:57:48.387 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.log_readahead_size: 0 2021-03-09T08:57:48.387 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-09T08:57:48.388 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_background_jobs: 2 2021-03-09T08:57:48.388 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_background_compactions: -1 2021-03-09T08:57:48.388 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-09T08:57:48.388 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-09T08:57:48.388 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-09T08:57:48.389 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-09T08:57:48.389 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-09T08:57:48.389 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-09T08:57:48.389 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-09T08:57:48.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-09T08:57:48.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_open_files: -1 2021-03-09T08:57:48.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-09T08:57:48.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-09T08:57:48.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-09T08:57:48.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-09T08:57:48.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Compression algorithms supported: 2021-03-09T08:57:48.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-09T08:57:48.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: kZSTD supported: 0 2021-03-09T08:57:48.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: kXpressCompression supported: 0 2021-03-09T08:57:48.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-09T08:57:48.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: kLZ4Compression supported: 1 2021-03-09T08:57:48.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: kBZip2Compression supported: 0 2021-03-09T08:57:48.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: kZlibCompression supported: 1 2021-03-09T08:57:48.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: kSnappyCompression supported: 1 2021-03-09T08:57:48.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-09T08:57:48.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 2021-03-09T08:57:48.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-09T08:57:48.394 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-09T08:57:48.394 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.merge_operator: 2021-03-09T08:57:48.394 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compaction_filter: None 2021-03-09T08:57:48.394 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-09T08:57:48.395 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-09T08:57:48.395 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-09T08:57:48.395 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55adca312230) 2021-03-09T08:57:48.395 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: cache_index_and_filter_blocks: 1 2021-03-09T08:57:48.396 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-09T08:57:48.396 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-09T08:57:48.396 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: pin_top_level_index_and_filter: 1 2021-03-09T08:57:48.396 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: index_type: 0 2021-03-09T08:57:48.396 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: data_block_index_type: 0 2021-03-09T08:57:48.397 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: index_shortening: 1 2021-03-09T08:57:48.397 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: data_block_hash_table_util_ratio: 0.750000 2021-03-09T08:57:48.397 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: hash_index_allow_collision: 1 2021-03-09T08:57:48.397 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: checksum: 1 2021-03-09T08:57:48.398 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: no_block_cache: 0 2021-03-09T08:57:48.398 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: block_cache: 0x55add254a510 2021-03-09T08:57:48.398 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: block_cache_name: BinnedLRUCache 2021-03-09T08:57:48.398 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: block_cache_options: 2021-03-09T08:57:48.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: capacity : 536870912 2021-03-09T08:57:48.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: num_shard_bits : 4 2021-03-09T08:57:48.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: strict_capacity_limit : 0 2021-03-09T08:57:48.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: high_pri_pool_ratio: 0.000 2021-03-09T08:57:48.400 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: block_cache_compressed: (nil) 2021-03-09T08:57:48.400 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: persistent_cache: (nil) 2021-03-09T08:57:48.400 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: block_size: 4096 2021-03-09T08:57:48.400 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: block_size_deviation: 10 2021-03-09T08:57:48.401 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: block_restart_interval: 16 2021-03-09T08:57:48.401 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: index_block_restart_interval: 1 2021-03-09T08:57:48.401 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: metadata_block_size: 4096 2021-03-09T08:57:48.401 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: partition_filters: 0 2021-03-09T08:57:48.401 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: use_delta_encoding: 1 2021-03-09T08:57:48.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-09T08:57:48.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: whole_key_filtering: 1 2021-03-09T08:57:48.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: verify_compression: 0 2021-03-09T08:57:48.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: read_amp_bytes_per_bit: 0 2021-03-09T08:57:48.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: format_version: 2 2021-03-09T08:57:48.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: enable_index_compression: 1 2021-03-09T08:57:48.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: block_align: 0 2021-03-09T08:57:48.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-09T08:57:48.404 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-09T08:57:48.404 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compression: NoCompression 2021-03-09T08:57:48.404 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-09T08:57:48.404 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-09T08:57:48.404 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-09T08:57:48.405 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.num_levels: 7 2021-03-09T08:57:48.405 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-09T08:57:48.405 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-09T08:57:48.405 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-09T08:57:48.406 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-09T08:57:48.406 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-09T08:57:48.406 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-09T08:57:48.406 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-09T08:57:48.407 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-09T08:57:48.407 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-09T08:57:48.407 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-09T08:57:48.407 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-09T08:57:48.407 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-09T08:57:48.408 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-09T08:57:48.408 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-09T08:57:48.408 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-09T08:57:48.408 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-09T08:57:48.409 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-09T08:57:48.409 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-09T08:57:48.409 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-09T08:57:48.409 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-09T08:57:48.410 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-09T08:57:48.410 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-09T08:57:48.410 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-09T08:57:48.410 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-09T08:57:48.410 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-09T08:57:48.411 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-09T08:57:48.411 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-09T08:57:48.411 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-09T08:57:48.411 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-09T08:57:48.411 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-09T08:57:48.412 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-09T08:57:48.412 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-09T08:57:48.412 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-09T08:57:48.412 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-09T08:57:48.413 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-09T08:57:48.413 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-09T08:57:48.413 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-09T08:57:48.413 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-09T08:57:48.414 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-09T08:57:48.414 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-09T08:57:48.414 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-09T08:57:48.414 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-09T08:57:48.414 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-09T08:57:48.415 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-09T08:57:48.415 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-09T08:57:48.415 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-09T08:57:48.415 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-09T08:57:48.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.table_properties_collectors: 2021-03-09T08:57:48.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.inplace_update_support: 0 2021-03-09T08:57:48.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-09T08:57:48.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-09T08:57:48.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-09T08:57:48.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-09T08:57:48.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.bloom_locality: 0 2021-03-09T08:57:48.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.max_successive_merges: 0 2021-03-09T08:57:48.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-09T08:57:48.418 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-09T08:57:48.418 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-09T08:57:48.418 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-09T08:57:48.418 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.ttl: 2592000 2021-03-09T08:57:48.419 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.935+0000 7f202a18b700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-09T08:57:48.419 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.939+0000 7f202a18b700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-09T08:57:48.419 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.939+0000 7f202a18b700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-03-09T08:57:48.419 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.939+0000 7f202a18b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280267940827, "job": 1, "event": "recovery_started", "log_files": [3]} 2021-03-09T08:57:48.419 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.939+0000 7f202a18b700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2021-03-09T08:57:48.420 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.939+0000 7f202a18b700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-09T08:57:48.420 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.939+0000 7f202a18b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280267942319, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1615280267, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-09T08:57:48.420 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.939+0000 7f202a18b700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2021-03-09T08:57:48.420 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.939+0000 7f202a18b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280267943395, "job": 1, "event": "recovery_finished"} 2021-03-09T08:57:48.421 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.943+0000 7f202a18b700 4 rocksdb: DB pointer 0x55add260f800 2021-03-09T08:57:48.421 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.943+0000 7f202a18b700 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2021-03-09T08:57:48.421 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.943+0000 7f202a18b700 0 using public_addrv [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2021-03-09T08:57:48.421 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.943+0000 7f201919b700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-09T08:57:48.421 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.943+0000 7f201919b700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-09T08:57:48.422 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: ** DB Stats ** 2021-03-09T08:57:48.422 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T08:57:48.422 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-09T08:57:48.422 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-09T08:57:48.423 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T08:57:48.423 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-09T08:57:48.423 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-09T08:57:48.423 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T08:57:48.424 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: ** Compaction Stats [default] ** 2021-03-09T08:57:48.424 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T08:57:48.424 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T08:57:48.424 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.0 0.00 0.00 1 0.001 0 0 2021-03-09T08:57:48.424 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.0 0.00 0.00 1 0.001 0 0 2021-03-09T08:57:48.425 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.0 0.00 0.00 1 0.001 0 0 2021-03-09T08:57:48.425 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: ** Compaction Stats [default] ** 2021-03-09T08:57:48.425 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T08:57:48.425 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T08:57:48.426 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.00 0.00 1 0.001 0 0 2021-03-09T08:57:48.426 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T08:57:48.426 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T08:57:48.426 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T08:57:48.426 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T08:57:48.427 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T08:57:48.427 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T08:57:48.427 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Cumulative compaction: 0.00 GB write, 0.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T08:57:48.427 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Interval compaction: 0.00 GB write, 0.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T08:57:48.428 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T08:57:48.428 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T08:57:48.428 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: ** Compaction Stats [default] ** 2021-03-09T08:57:48.428 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T08:57:48.429 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T08:57:48.429 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.0 0.00 0.00 1 0.001 0 0 2021-03-09T08:57:48.429 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.0 0.00 0.00 1 0.001 0 0 2021-03-09T08:57:48.429 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T08:57:48.430 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: ** Compaction Stats [default] ** 2021-03-09T08:57:48.430 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T08:57:48.430 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T08:57:48.430 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.00 0.00 1 0.001 0 0 2021-03-09T08:57:48.430 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T08:57:48.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T08:57:48.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T08:57:48.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T08:57:48.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T08:57:48.432 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T08:57:48.432 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Cumulative compaction: 0.00 GB write, 0.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T08:57:48.432 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T08:57:48.432 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T08:57:48.432 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T08:57:48.433 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.943+0000 7f202a18b700 0 starting mon.c rank -1 at public addrs [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] at bind addrs [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:57:48.433 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.943+0000 7f202a18b700 1 mon.c@-1(???) e0 preinit fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:57:48.433 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.967+0000 7f201b9a0700 0 mon.c@-1(synchronizing).mds e1 new map 2021-03-09T08:57:48.433 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.967+0000 7f201b9a0700 0 mon.c@-1(synchronizing).mds e1 print_map 2021-03-09T08:57:48.434 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: e1 2021-03-09T08:57:48.434 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: enable_multiple, ever_enabled_multiple: 1,1 2021-03-09T08:57:48.434 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-03-09T08:57:48.434 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: legacy client fscid: -1 2021-03-09T08:57:48.434 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: 2021-03-09T08:57:48.435 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: No filesystems configured 2021-03-09T08:57:48.435 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.967+0000 7f201b9a0700 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2021-03-09T08:57:48.435 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.967+0000 7f201b9a0700 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2021-03-09T08:57:48.435 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.967+0000 7f201b9a0700 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2021-03-09T08:57:48.435 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.967+0000 7f201b9a0700 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2021-03-09T08:57:48.436 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.967+0000 7f201b9a0700 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2021-03-09T08:57:48.436 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.967+0000 7f201b9a0700 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2021-03-09T08:57:48.436 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.967+0000 7f201b9a0700 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2021-03-09T08:57:48.436 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.967+0000 7f201b9a0700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T08:57:48.437 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.967+0000 7f201b9a0700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T08:57:48.437 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.967+0000 7f201b9a0700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T08:57:48.437 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: cluster 2021-03-09T08:57:45.286377+0000 mgr.y (mgr.14144) 32 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:48.437 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: audit 2021-03-09T08:57:46.309137+0000 mon.a (mon.0) 106 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:57:48.437 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: audit 2021-03-09T08:57:46.312547+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:57:48.438 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:47 smithi005 bash[16589]: debug 2021-03-09T08:57:47.967+0000 7f201b9a0700 1 mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2021-03-09T08:57:53.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: cluster 2021-03-09T08:57:47.286909+0000 mgr.y (mgr.14144) 33 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:53.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: audit 2021-03-09T08:57:47.980226+0000 mon.a (mon.0) 109 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-03-09T08:57:53.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: cluster 2021-03-09T08:57:47.980616+0000 mon.a (mon.0) 110 : cluster [INF] mon.a calling monitor election 2021-03-09T08:57:53.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: audit 2021-03-09T08:57:47.982205+0000 mon.a (mon.0) 111 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-09T08:57:53.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: audit 2021-03-09T08:57:48.976342+0000 mon.a (mon.0) 112 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-09T08:57:53.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: cluster 2021-03-09T08:57:49.287370+0000 mgr.y (mgr.14144) 34 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:53.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: audit 2021-03-09T08:57:49.976296+0000 mon.a (mon.0) 113 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-09T08:57:53.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: cluster 2021-03-09T08:57:49.978789+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2021-03-09T08:57:53.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: audit 2021-03-09T08:57:50.975881+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-09T08:57:53.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: cluster 2021-03-09T08:57:51.287852+0000 mgr.y (mgr.14144) 35 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:53.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: audit 2021-03-09T08:57:51.976556+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-09T08:57:53.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: audit 2021-03-09T08:57:52.976654+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-09T08:57:53.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: cluster 2021-03-09T08:57:53.005518+0000 mon.a (mon.0) 117 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-03-09T08:57:53.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: cluster 2021-03-09T08:57:53.011339+0000 mon.a (mon.0) 118 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2021-03-09T08:57:53.373 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: cluster 2021-03-09T08:57:53.011488+0000 mon.a (mon.0) 119 : cluster [DBG] fsmap 2021-03-09T08:57:53.373 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: cluster 2021-03-09T08:57:53.011585+0000 mon.a (mon.0) 120 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-09T08:57:53.373 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: cluster 2021-03-09T08:57:53.012020+0000 mon.a (mon.0) 121 : cluster [DBG] mgrmap e13: y(active, since 67s) 2021-03-09T08:57:53.373 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: cluster 2021-03-09T08:57:53.018975+0000 mon.a (mon.0) 122 : cluster [INF] overall HEALTH_OK 2021-03-09T08:57:53.374 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: audit 2021-03-09T08:57:53.023835+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:57:53.374 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[12738]: audit 2021-03-09T08:57:53.030313+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:57:53.375 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: cluster 2021-03-09T08:57:47.286909+0000 mgr.y (mgr.14144) 33 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:53.375 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: audit 2021-03-09T08:57:47.980226+0000 mon.a (mon.0) 109 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-03-09T08:57:53.375 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: cluster 2021-03-09T08:57:47.980616+0000 mon.a (mon.0) 110 : cluster [INF] mon.a calling monitor election 2021-03-09T08:57:53.375 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: audit 2021-03-09T08:57:47.982205+0000 mon.a (mon.0) 111 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-09T08:57:53.376 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: audit 2021-03-09T08:57:48.976342+0000 mon.a (mon.0) 112 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-09T08:57:53.376 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: cluster 2021-03-09T08:57:49.287370+0000 mgr.y (mgr.14144) 34 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:53.376 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: audit 2021-03-09T08:57:49.976296+0000 mon.a (mon.0) 113 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-09T08:57:53.376 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: cluster 2021-03-09T08:57:49.978789+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2021-03-09T08:57:53.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: audit 2021-03-09T08:57:50.975881+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-09T08:57:53.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: cluster 2021-03-09T08:57:51.287852+0000 mgr.y (mgr.14144) 35 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:53.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: audit 2021-03-09T08:57:51.976556+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-09T08:57:53.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: audit 2021-03-09T08:57:52.976654+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-09T08:57:53.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: cluster 2021-03-09T08:57:53.005518+0000 mon.a (mon.0) 117 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-03-09T08:57:53.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: cluster 2021-03-09T08:57:53.011339+0000 mon.a (mon.0) 118 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2021-03-09T08:57:53.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: cluster 2021-03-09T08:57:53.011488+0000 mon.a (mon.0) 119 : cluster [DBG] fsmap 2021-03-09T08:57:53.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: cluster 2021-03-09T08:57:53.011585+0000 mon.a (mon.0) 120 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-09T08:57:53.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: cluster 2021-03-09T08:57:53.012020+0000 mon.a (mon.0) 121 : cluster [DBG] mgrmap e13: y(active, since 67s) 2021-03-09T08:57:53.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: cluster 2021-03-09T08:57:53.018975+0000 mon.a (mon.0) 122 : cluster [INF] overall HEALTH_OK 2021-03-09T08:57:53.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: audit 2021-03-09T08:57:53.023835+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:57:53.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:53 smithi005 bash[16589]: audit 2021-03-09T08:57:53.030313+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:57:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.019+0000 7ff69bfff700 1 -- 172.21.15.5:0/1416059147 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff688003250 msgr2=0x7ff6880056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:57:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.019+0000 7ff69bfff700 1 --2- 172.21.15.5:0/1416059147 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff688003250 0x7ff6880056e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7ff69c0f57c0 tx=0x7ff68c00b040).stop 2021-03-09T08:57:54.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.023+0000 7ff69bfff700 1 -- 172.21.15.5:0/1416059147 shutdown_connections 2021-03-09T08:57:54.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.023+0000 7ff69bfff700 1 --2- 172.21.15.5:0/1416059147 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff688003250 0x7ff6880056e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:54.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.023+0000 7ff69bfff700 1 -- 172.21.15.5:0/1416059147 >> 172.21.15.5:0/1416059147 conn(0x7ff69c0f0170 msgr2=0x7ff69c0f2590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:57:54.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.023+0000 7ff69bfff700 1 -- 172.21.15.5:0/1416059147 shutdown_connections 2021-03-09T08:57:54.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.023+0000 7ff69bfff700 1 -- 172.21.15.5:0/1416059147 wait complete. 2021-03-09T08:57:54.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.023+0000 7ff69bfff700 1 Processor -- start 2021-03-09T08:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.023+0000 7ff69bfff700 1 -- start start 2021-03-09T08:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.023+0000 7ff69bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff688003250 0x7ff69c107d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.023+0000 7ff69bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff69c05fb10 0x7ff69c10a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.023+0000 7ff69bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c10a7f0 con 0x7ff688003250 2021-03-09T08:57:54.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.023+0000 7ff69bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c10a930 con 0x7ff69c05fb10 2021-03-09T08:57:54.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.023+0000 7ff69affd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff688003250 0x7ff69c107d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:54.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.023+0000 7ff69affd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff688003250 0x7ff69c107d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46540/0 (socket says 172.21.15.5:46540) 2021-03-09T08:57:54.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.023+0000 7ff69affd700 1 -- 172.21.15.5:0/3231851814 learned_addr learned my addr 172.21.15.5:0/3231851814 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:57:54.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.023+0000 7ff693fff700 1 --2- 172.21.15.5:0/3231851814 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff69c05fb10 0x7ff69c10a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:54.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.027+0000 7ff693fff700 1 -- 172.21.15.5:0/3231851814 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff69c05fb10 msgr2=0x7ff69c10a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T08:57:54.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.027+0000 7ff693fff700 1 -- 172.21.15.5:0/3231851814 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff69c05fb10 msgr2=0x7ff69c10a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:54.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.027+0000 7ff693fff700 1 --2- 172.21.15.5:0/3231851814 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff69c05fb10 0x7ff69c10a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:54.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.027+0000 7ff693fff700 1 --2- 172.21.15.5:0/3231851814 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff69c05fb10 0x7ff69c10a220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:54.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.027+0000 7ff69affd700 1 -- 172.21.15.5:0/3231851814 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff69c05fb10 msgr2=0x7ff69c10a220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:54.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.027+0000 7ff69affd700 1 --2- 172.21.15.5:0/3231851814 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff69c05fb10 0x7ff69c10a220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:54.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.027+0000 7ff69affd700 1 -- 172.21.15.5:0/3231851814 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69c10aaf0 con 0x7ff688003250 2021-03-09T08:57:54.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.027+0000 7ff69affd700 1 --2- 172.21.15.5:0/3231851814 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff688003250 0x7ff69c107d20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7ff68c009880 tx=0x7ff68c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:54.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.027+0000 7ff6937fe700 1 -- 172.21.15.5:0/3231851814 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7ff68c025070 con 0x7ff688003250 2021-03-09T08:57:54.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.027+0000 7ff6937fe700 1 -- 172.21.15.5:0/3231851814 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7ff68c004e90 con 0x7ff688003250 2021-03-09T08:57:54.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.027+0000 7ff69bfff700 1 -- 172.21.15.5:0/3231851814 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff69c10ad40 con 0x7ff688003250 2021-03-09T08:57:54.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.027+0000 7ff6937fe700 1 -- 172.21.15.5:0/3231851814 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7ff68c021b60 con 0x7ff688003250 2021-03-09T08:57:54.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.027+0000 7ff69bfff700 1 -- 172.21.15.5:0/3231851814 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69c10b2a0 con 0x7ff688003250 2021-03-09T08:57:54.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.027+0000 7ff6937fe700 1 -- 172.21.15.5:0/3231851814 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 13) v1 ==== 36113+0+0 (secure 0 0 0) 0x7ff68c02a410 con 0x7ff688003250 2021-03-09T08:57:54.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.027+0000 7ff6937fe700 1 --2- 172.21.15.5:0/3231851814 >> v2:172.21.15.5:6800/4269212764 conn(0x7ff67c030000 0x7ff67c0324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:54.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.027+0000 7ff6937fe700 1 -- 172.21.15.5:0/3231851814 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7ff68c0422c0 con 0x7ff688003250 2021-03-09T08:57:54.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.031+0000 7ff693fff700 1 --2- 172.21.15.5:0/3231851814 >> v2:172.21.15.5:6800/4269212764 conn(0x7ff67c030000 0x7ff67c0324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:54.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.031+0000 7ff69bfff700 1 -- 172.21.15.5:0/3231851814 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69c0589f0 con 0x7ff688003250 2021-03-09T08:57:54.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.031+0000 7ff693fff700 1 --2- 172.21.15.5:0/3231851814 >> v2:172.21.15.5:6800/4269212764 conn(0x7ff67c030000 0x7ff67c0324c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7ff69c05f290 tx=0x7ff684008040).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:54.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.039+0000 7ff6937fe700 1 -- 172.21.15.5:0/3231851814 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7ff68c01b070 con 0x7ff688003250 2021-03-09T08:57:54.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.267+0000 7ff69bfff700 1 -- 172.21.15.5:0/3231851814 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff69c0f7650 con 0x7ff688003250 2021-03-09T08:57:54.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.267+0000 7ff6937fe700 1 -- 172.21.15.5:0/3231851814 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+948 (secure 0 0 0) 0x7ff68c01e2f0 con 0x7ff688003250 2021-03-09T08:57:54.272 INFO:teuthology.orchestra.run.smithi005.stdout: 2021-03-09T08:57:54.272 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":2,"fsid":"252abbe0-80b5-11eb-907b-001a4aab830c","modified":"2021-03-09T08:57:47.976769Z","created":"2021-03-09T08:55:39.986925Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2021-03-09T08:57:54.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.271+0000 7ff69bfff700 1 -- 172.21.15.5:0/3231851814 >> v2:172.21.15.5:6800/4269212764 conn(0x7ff67c030000 msgr2=0x7ff67c0324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:57:54.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.271+0000 7ff69bfff700 1 --2- 172.21.15.5:0/3231851814 >> v2:172.21.15.5:6800/4269212764 conn(0x7ff67c030000 0x7ff67c0324c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7ff69c05f290 tx=0x7ff684008040).stop 2021-03-09T08:57:54.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.271+0000 7ff69bfff700 1 -- 172.21.15.5:0/3231851814 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff688003250 msgr2=0x7ff69c107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:57:54.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.271+0000 7ff69bfff700 1 --2- 172.21.15.5:0/3231851814 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff688003250 0x7ff69c107d20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7ff68c009880 tx=0x7ff68c00b040).stop 2021-03-09T08:57:54.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.271+0000 7ff69bfff700 1 -- 172.21.15.5:0/3231851814 shutdown_connections 2021-03-09T08:57:54.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.271+0000 7ff69bfff700 1 --2- 172.21.15.5:0/3231851814 >> v2:172.21.15.5:6800/4269212764 conn(0x7ff67c030000 0x7ff67c0324c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:54.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.271+0000 7ff69bfff700 1 --2- 172.21.15.5:0/3231851814 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff688003250 0x7ff69c107d20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:54.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.271+0000 7ff69bfff700 1 --2- 172.21.15.5:0/3231851814 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff69c05fb10 0x7ff69c10a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:54.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.271+0000 7ff69bfff700 1 -- 172.21.15.5:0/3231851814 >> 172.21.15.5:0/3231851814 conn(0x7ff69c0f0170 msgr2=0x7ff69c0f88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:57:54.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.271+0000 7ff69bfff700 1 -- 172.21.15.5:0/3231851814 shutdown_connections 2021-03-09T08:57:54.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:57:54.271+0000 7ff69bfff700 1 -- 172.21.15.5:0/3231851814 wait complete. 2021-03-09T08:57:54.277 INFO:teuthology.orchestra.run.smithi005.stderr:dumped monmap epoch 2 2021-03-09T08:57:54.335 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:54 smithi005 bash[12738]: cluster 2021-03-09T08:57:53.288305+0000 mgr.y (mgr.14144) 36 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:54.335 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:54 smithi005 bash[12738]: audit 2021-03-09T08:57:53.976823+0000 mon.a (mon.0) 125 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-09T08:57:54.335 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:54 smithi005 bash[16589]: cluster 2021-03-09T08:57:53.288305+0000 mgr.y (mgr.14144) 36 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:54.336 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:54 smithi005 bash[16589]: audit 2021-03-09T08:57:53.976823+0000 mon.a (mon.0) 125 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-09T08:57:55.087 INFO:tasks.cephadm:Adding mon.b on smithi146 2021-03-09T08:57:55.087 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph orch daemon add mon smithi146:172.21.15.146=b 2021-03-09T08:57:55.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:55 smithi005 bash[12738]: audit 2021-03-09T08:57:54.270823+0000 mon.a (mon.0) 126 : audit [DBG] from='client.? 172.21.15.5:0/3231851814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-09T08:57:55.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:55 smithi005 bash[16589]: audit 2021-03-09T08:57:54.270823+0000 mon.a (mon.0) 126 : audit [DBG] from='client.? 172.21.15.5:0/3231851814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-09T08:57:56.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:56 smithi005 bash[12738]: cluster 2021-03-09T08:57:55.288762+0000 mgr.y (mgr.14144) 37 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:56.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:56 smithi005 bash[16589]: cluster 2021-03-09T08:57:55.288762+0000 mgr.y (mgr.14144) 37 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:57.911 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.906+0000 7fe2673cc700 1 -- 172.21.15.146:0/4199642880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe26005faf0 msgr2=0x7fe2600fab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:57:57.911 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.906+0000 7fe2673cc700 1 --2- 172.21.15.146:0/4199642880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe26005faf0 0x7fe2600fab90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fe25c004d00 tx=0x7fe25c009b30).stop 2021-03-09T08:57:57.912 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.906+0000 7fe2673cc700 1 -- 172.21.15.146:0/4199642880 shutdown_connections 2021-03-09T08:57:57.912 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.906+0000 7fe2673cc700 1 --2- 172.21.15.146:0/4199642880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe26005faf0 0x7fe2600fab90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:57.912 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.906+0000 7fe2673cc700 1 -- 172.21.15.146:0/4199642880 >> 172.21.15.146:0/4199642880 conn(0x7fe2600f0170 msgr2=0x7fe2600f25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:57:57.912 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.906+0000 7fe2673cc700 1 -- 172.21.15.146:0/4199642880 shutdown_connections 2021-03-09T08:57:57.913 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.906+0000 7fe2673cc700 1 -- 172.21.15.146:0/4199642880 wait complete. 2021-03-09T08:57:57.915 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.906+0000 7fe2673cc700 1 Processor -- start 2021-03-09T08:57:57.916 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.906+0000 7fe2673cc700 1 -- start start 2021-03-09T08:57:57.916 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.906+0000 7fe2673cc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe26005faf0 0x7fe2601072c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:57.916 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.906+0000 7fe2673cc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2601077b0 0x7fe2601086a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:57.916 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.906+0000 7fe2673cc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe260107d10 con 0x7fe26005faf0 2021-03-09T08:57:57.917 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.906+0000 7fe2673cc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe260107e50 con 0x7fe2601077b0 2021-03-09T08:57:57.917 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.906+0000 7fe2663ca700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe26005faf0 0x7fe2601072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:57.917 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.906+0000 7fe265bc9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2601077b0 0x7fe2601086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:57.917 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe2663ca700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe26005faf0 0x7fe2601072c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.146:50974/0 (socket says 172.21.15.146:50974) 2021-03-09T08:57:57.918 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe2663ca700 1 -- 172.21.15.146:0/241371650 learned_addr learned my addr 172.21.15.146:0/241371650 (peer_addr_for_me v2:172.21.15.146:0/0) 2021-03-09T08:57:57.918 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe265bc9700 1 -- 172.21.15.146:0/241371650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2601077b0 msgr2=0x7fe2601086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T08:57:57.918 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe265bc9700 1 -- 172.21.15.146:0/241371650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2601077b0 msgr2=0x7fe2601086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:57.918 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe265bc9700 1 --2- 172.21.15.146:0/241371650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2601077b0 0x7fe2601086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:57.918 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe265bc9700 1 --2- 172.21.15.146:0/241371650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2601077b0 0x7fe2601086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:57.919 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe2663ca700 1 -- 172.21.15.146:0/241371650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2601077b0 msgr2=0x7fe2601086a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:57.919 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe2663ca700 1 --2- 172.21.15.146:0/241371650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2601077b0 0x7fe2601086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:57.919 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe2663ca700 1 -- 172.21.15.146:0/241371650 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe260108b90 con 0x7fe26005faf0 2021-03-09T08:57:57.919 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe2663ca700 1 --2- 172.21.15.146:0/241371650 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe26005faf0 0x7fe2601072c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fe25c00bed0 tx=0x7fe25c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:57.919 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe252ffd700 1 -- 172.21.15.146:0/241371650 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fe25c00e030 con 0x7fe26005faf0 2021-03-09T08:57:57.920 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe252ffd700 1 -- 172.21.15.146:0/241371650 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7fe25c013c50 con 0x7fe26005faf0 2021-03-09T08:57:57.920 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe252ffd700 1 -- 172.21.15.146:0/241371650 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fe25c018070 con 0x7fe26005faf0 2021-03-09T08:57:57.920 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe2673cc700 1 -- 172.21.15.146:0/241371650 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe260108de0 con 0x7fe26005faf0 2021-03-09T08:57:57.920 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe2673cc700 1 -- 172.21.15.146:0/241371650 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2601092c0 con 0x7fe26005faf0 2021-03-09T08:57:57.920 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe252ffd700 1 -- 172.21.15.146:0/241371650 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 13) v1 ==== 36113+0+0 (secure 0 0 0) 0x7fe25c0137d0 con 0x7fe26005faf0 2021-03-09T08:57:57.921 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe2673cc700 1 -- 172.21.15.146:0/241371650 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2600589f0 con 0x7fe26005faf0 2021-03-09T08:57:57.922 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe252ffd700 1 --2- 172.21.15.146:0/241371650 >> v2:172.21.15.5:6800/4269212764 conn(0x7fe24c030000 0x7fe24c0324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:57.922 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe252ffd700 1 -- 172.21.15.146:0/241371650 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7fe25c041ab0 con 0x7fe26005faf0 2021-03-09T08:57:57.922 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.910+0000 7fe265bc9700 1 --2- 172.21.15.146:0/241371650 >> v2:172.21.15.5:6800/4269212764 conn(0x7fe24c030000 0x7fe24c0324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:57.922 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.914+0000 7fe265bc9700 1 --2- 172.21.15.146:0/241371650 >> v2:172.21.15.5:6800/4269212764 conn(0x7fe24c030000 0x7fe24c0324c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fe254006c30 tx=0x7fe254008040).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:57.929 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:57.922+0000 7fe252ffd700 1 -- 172.21.15.146:0/241371650 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7fe25c043e60 con 0x7fe26005faf0 2021-03-09T08:57:58.124 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:57:58.118+0000 7fe2673cc700 1 -- 172.21.15.146:0/241371650 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi146:172.21.15.146=b", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2600f1170 con 0x7fe24c030000 2021-03-09T08:57:59.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:58 smithi005 bash[12738]: cluster 2021-03-09T08:57:57.289185+0000 mgr.y (mgr.14144) 38 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:59.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:58 smithi005 bash[12738]: audit 2021-03-09T08:57:58.127383+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-03-09T08:57:59.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:58 smithi005 bash[12738]: audit 2021-03-09T08:57:58.129427+0000 mon.a (mon.0) 128 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:57:59.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:58 smithi005 bash[16589]: cluster 2021-03-09T08:57:57.289185+0000 mgr.y (mgr.14144) 38 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:57:59.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:58 smithi005 bash[16589]: audit 2021-03-09T08:57:58.127383+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-03-09T08:57:59.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:58 smithi005 bash[16589]: audit 2021-03-09T08:57:58.129427+0000 mon.a (mon.0) 128 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:00.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:59 smithi005 bash[12738]: audit 2021-03-09T08:57:58.124739+0000 mgr.y (mgr.14144) 39 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi146:172.21.15.146=b", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:58:00.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:57:59 smithi005 bash[12738]: cephadm 2021-03-09T08:57:58.131025+0000 mgr.y (mgr.14144) 40 : cephadm [INF] Deploying daemon mon.b on smithi146 2021-03-09T08:58:00.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:59 smithi005 bash[16589]: audit 2021-03-09T08:57:58.124739+0000 mgr.y (mgr.14144) 39 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi146:172.21.15.146=b", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:58:00.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:57:59 smithi005 bash[16589]: cephadm 2021-03-09T08:57:58.131025+0000 mgr.y (mgr.14144) 40 : cephadm [INF] Deploying daemon mon.b on smithi146 2021-03-09T08:58:01.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:00 smithi005 bash[12738]: cluster 2021-03-09T08:57:59.289667+0000 mgr.y (mgr.14144) 41 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:01.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:00 smithi005 bash[16589]: cluster 2021-03-09T08:57:59.289667+0000 mgr.y (mgr.14144) 41 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:02.800 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:02.794+0000 7fe252ffd700 1 -- 172.21.15.146:0/241371650 <== mgr.14144 v2:172.21.15.5:6800/4269212764 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+35 (secure 0 0 0) 0x7fe2600f1170 con 0x7fe24c030000 2021-03-09T08:58:02.800 INFO:teuthology.orchestra.run.smithi146.stdout:Deployed mon.b on host 'smithi146' 2021-03-09T08:58:02.803 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:02.798+0000 7fe2673cc700 1 -- 172.21.15.146:0/241371650 >> v2:172.21.15.5:6800/4269212764 conn(0x7fe24c030000 msgr2=0x7fe24c0324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:02.803 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:02.798+0000 7fe2673cc700 1 --2- 172.21.15.146:0/241371650 >> v2:172.21.15.5:6800/4269212764 conn(0x7fe24c030000 0x7fe24c0324c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fe254006c30 tx=0x7fe254008040).stop 2021-03-09T08:58:02.804 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:02.798+0000 7fe2673cc700 1 -- 172.21.15.146:0/241371650 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe26005faf0 msgr2=0x7fe2601072c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:02.804 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:02.798+0000 7fe2673cc700 1 --2- 172.21.15.146:0/241371650 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe26005faf0 0x7fe2601072c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fe25c00bed0 tx=0x7fe25c009b30).stop 2021-03-09T08:58:02.804 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:02.798+0000 7fe2673cc700 1 -- 172.21.15.146:0/241371650 shutdown_connections 2021-03-09T08:58:02.804 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:02.798+0000 7fe2673cc700 1 --2- 172.21.15.146:0/241371650 >> v2:172.21.15.5:6800/4269212764 conn(0x7fe24c030000 0x7fe24c0324c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:02.805 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:02.798+0000 7fe2673cc700 1 --2- 172.21.15.146:0/241371650 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe26005faf0 0x7fe2601072c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:02.805 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:02.798+0000 7fe2673cc700 1 --2- 172.21.15.146:0/241371650 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2601077b0 0x7fe2601086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:02.805 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:02.798+0000 7fe2673cc700 1 -- 172.21.15.146:0/241371650 >> 172.21.15.146:0/241371650 conn(0x7fe2600f0170 msgr2=0x7fe2600f0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:58:02.805 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:02.798+0000 7fe2673cc700 1 -- 172.21.15.146:0/241371650 shutdown_connections 2021-03-09T08:58:02.805 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:02.798+0000 7fe2673cc700 1 -- 172.21.15.146:0/241371650 wait complete. 2021-03-09T08:58:03.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:02 smithi005 bash[12738]: cluster 2021-03-09T08:58:01.290090+0000 mgr.y (mgr.14144) 42 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:03.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:02 smithi005 bash[12738]: audit 2021-03-09T08:58:02.792770+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:03.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:02 smithi005 bash[12738]: audit 2021-03-09T08:58:02.797694+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:03.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:02 smithi005 bash[16589]: cluster 2021-03-09T08:58:01.290090+0000 mgr.y (mgr.14144) 42 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:03.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:02 smithi005 bash[16589]: audit 2021-03-09T08:58:02.792770+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:03.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:02 smithi005 bash[16589]: audit 2021-03-09T08:58:02.797694+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:04.045 DEBUG:teuthology.orchestra.run.smithi146:mon.b> sudo journalctl -f -n 0 -u ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.b.service 2021-03-09T08:58:04.057 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2021-03-09T08:58:04.057 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph mon dump -f json 2021-03-09T08:58:04.112 INFO:journalctl@ceph.mon.b.smithi146.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-09T08:58:04.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.326+0000 7fa56c31d700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-09T08:58:04.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.326+0000 7fa56c31d700 0 ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev), process ceph-mon, pid 7 2021-03-09T08:58:04.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.326+0000 7fa56c31d700 0 pidfile_write: ignore empty --pid-file 2021-03-09T08:58:04.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.330+0000 7fa56c31d700 0 load: jerasure load: lrc load: isa 2021-03-09T08:58:04.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: RocksDB version: 6.8.1 2021-03-09T08:58:04.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-09T08:58:04.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Compile date Mar 9 2021 2021-03-09T08:58:04.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: DB SUMMARY 2021-03-09T08:58:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: CURRENT file: CURRENT 2021-03-09T08:58:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-09T08:58:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2021-03-09T08:58:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2021-03-09T08:58:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000003.log size: 636 ; 2021-03-09T08:58:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.error_if_exists: 0 2021-03-09T08:58:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.create_if_missing: 0 2021-03-09T08:58:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.paranoid_checks: 1 2021-03-09T08:58:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.env: 0x56432f8ef1c0 2021-03-09T08:58:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.fs: Posix File System 2021-03-09T08:58:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.info_log: 0x564338c73120 2021-03-09T08:58:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-09T08:58:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.statistics: (nil) 2021-03-09T08:58:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.use_fsync: 0 2021-03-09T08:58:04.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_log_file_size: 0 2021-03-09T08:58:04.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-09T08:58:04.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-09T08:58:04.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-09T08:58:04.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-09T08:58:04.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.allow_fallocate: 1 2021-03-09T08:58:04.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-09T08:58:04.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-09T08:58:04.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.use_direct_reads: 0 2021-03-09T08:58:04.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-09T08:58:04.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-09T08:58:04.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.db_log_dir: 2021-03-09T08:58:04.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2021-03-09T08:58:04.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-09T08:58:04.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_subcompactions: 1 2021-03-09T08:58:04.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_background_flushes: -1 2021-03-09T08:58:04.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-09T08:58:04.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-09T08:58:04.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-09T08:58:04.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-09T08:58:04.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-09T08:58:04.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-09T08:58:04.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-09T08:58:04.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.write_buffer_manager: 0x564338cc7c50 2021-03-09T08:58:04.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-09T08:58:04.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-09T08:58:04.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-09T08:58:04.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-09T08:58:04.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-09T08:58:04.774 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-09T08:58:04.774 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-09T08:58:04.774 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-09T08:58:04.776 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-09T08:58:04.776 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.unordered_write: 0 2021-03-09T08:58:04.776 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-09T08:58:04.777 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-09T08:58:04.777 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-09T08:58:04.777 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-09T08:58:04.777 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.row_cache: None 2021-03-09T08:58:04.777 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.wal_filter: None 2021-03-09T08:58:04.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-09T08:58:04.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-09T08:58:04.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.preserve_deletes: 0 2021-03-09T08:58:04.778 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.two_write_queues: 0 2021-03-09T08:58:04.779 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-09T08:58:04.779 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.atomic_flush: 0 2021-03-09T08:58:04.779 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-09T08:58:04.779 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-09T08:58:04.779 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-09T08:58:04.780 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.log_readahead_size: 0 2021-03-09T08:58:04.780 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-09T08:58:04.780 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_background_jobs: 2 2021-03-09T08:58:04.780 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_background_compactions: -1 2021-03-09T08:58:04.780 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-09T08:58:04.781 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-09T08:58:04.781 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-09T08:58:04.781 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-09T08:58:04.781 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-09T08:58:04.781 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-09T08:58:04.782 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-09T08:58:04.782 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-09T08:58:04.782 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_open_files: -1 2021-03-09T08:58:04.783 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-09T08:58:04.783 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-09T08:58:04.783 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-09T08:58:04.783 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-09T08:58:04.783 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Compression algorithms supported: 2021-03-09T08:58:04.784 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-09T08:58:04.784 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: kZSTD supported: 0 2021-03-09T08:58:04.784 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: kXpressCompression supported: 0 2021-03-09T08:58:04.784 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-09T08:58:04.784 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: kLZ4Compression supported: 1 2021-03-09T08:58:04.785 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: kBZip2Compression supported: 0 2021-03-09T08:58:04.785 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: kZlibCompression supported: 1 2021-03-09T08:58:04.785 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: kSnappyCompression supported: 1 2021-03-09T08:58:04.785 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-09T08:58:04.786 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 2021-03-09T08:58:04.786 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-09T08:58:04.786 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-09T08:58:04.786 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.merge_operator: 2021-03-09T08:58:04.786 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compaction_filter: None 2021-03-09T08:58:04.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-09T08:58:04.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-09T08:58:04.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-09T08:58:04.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564330a16230) 2021-03-09T08:58:04.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: cache_index_and_filter_blocks: 1 2021-03-09T08:58:04.788 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-09T08:58:04.788 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-09T08:58:04.788 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: pin_top_level_index_and_filter: 1 2021-03-09T08:58:04.788 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: index_type: 0 2021-03-09T08:58:04.789 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: data_block_index_type: 0 2021-03-09T08:58:04.789 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: index_shortening: 1 2021-03-09T08:58:04.789 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: data_block_hash_table_util_ratio: 0.750000 2021-03-09T08:58:04.789 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: hash_index_allow_collision: 1 2021-03-09T08:58:04.789 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: checksum: 1 2021-03-09T08:58:04.790 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: no_block_cache: 0 2021-03-09T08:58:04.790 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: block_cache: 0x564338c4e510 2021-03-09T08:58:04.790 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: block_cache_name: BinnedLRUCache 2021-03-09T08:58:04.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: block_cache_options: 2021-03-09T08:58:04.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: capacity : 536870912 2021-03-09T08:58:04.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: num_shard_bits : 4 2021-03-09T08:58:04.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: strict_capacity_limit : 0 2021-03-09T08:58:04.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: high_pri_pool_ratio: 0.000 2021-03-09T08:58:04.792 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: block_cache_compressed: (nil) 2021-03-09T08:58:04.792 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: persistent_cache: (nil) 2021-03-09T08:58:04.792 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: block_size: 4096 2021-03-09T08:58:04.792 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: block_size_deviation: 10 2021-03-09T08:58:04.792 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: block_restart_interval: 16 2021-03-09T08:58:04.793 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: index_block_restart_interval: 1 2021-03-09T08:58:04.793 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: metadata_block_size: 4096 2021-03-09T08:58:04.793 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: partition_filters: 0 2021-03-09T08:58:04.793 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: use_delta_encoding: 1 2021-03-09T08:58:04.794 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-09T08:58:04.794 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: whole_key_filtering: 1 2021-03-09T08:58:04.794 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: verify_compression: 0 2021-03-09T08:58:04.794 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: read_amp_bytes_per_bit: 0 2021-03-09T08:58:04.795 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: format_version: 2 2021-03-09T08:58:04.795 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: enable_index_compression: 1 2021-03-09T08:58:04.795 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: block_align: 0 2021-03-09T08:58:04.795 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-09T08:58:04.796 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-09T08:58:04.796 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compression: NoCompression 2021-03-09T08:58:04.796 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-09T08:58:04.796 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-09T08:58:04.796 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-09T08:58:04.797 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.num_levels: 7 2021-03-09T08:58:04.797 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-09T08:58:04.797 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-09T08:58:04.797 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-09T08:58:04.797 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-09T08:58:04.798 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-09T08:58:04.798 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-09T08:58:04.798 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-09T08:58:04.798 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-09T08:58:04.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-09T08:58:04.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-09T08:58:04.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-09T08:58:04.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-09T08:58:04.799 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-09T08:58:04.800 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-09T08:58:04.800 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-09T08:58:04.800 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-09T08:58:04.800 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-09T08:58:04.800 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-09T08:58:04.801 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-09T08:58:04.801 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-09T08:58:04.801 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-09T08:58:04.801 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-09T08:58:04.801 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-09T08:58:04.802 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-09T08:58:04.802 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-09T08:58:04.802 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-09T08:58:04.802 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-09T08:58:04.802 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-09T08:58:04.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-09T08:58:04.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-09T08:58:04.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-09T08:58:04.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-09T08:58:04.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-09T08:58:04.804 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-09T08:58:04.804 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-09T08:58:04.804 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-09T08:58:04.804 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-09T08:58:04.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-09T08:58:04.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-09T08:58:04.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-09T08:58:04.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-09T08:58:04.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-09T08:58:04.806 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-09T08:58:04.806 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-09T08:58:04.806 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-09T08:58:04.806 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-09T08:58:04.806 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-09T08:58:04.807 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.table_properties_collectors: 2021-03-09T08:58:04.807 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.inplace_update_support: 0 2021-03-09T08:58:04.807 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-09T08:58:04.807 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-09T08:58:04.807 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-09T08:58:04.808 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-09T08:58:04.808 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.bloom_locality: 0 2021-03-09T08:58:04.808 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.max_successive_merges: 0 2021-03-09T08:58:04.808 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-09T08:58:04.808 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-09T08:58:04.809 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-09T08:58:04.809 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-09T08:58:04.809 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.ttl: 2592000 2021-03-09T08:58:04.809 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-09T08:58:04.809 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-09T08:58:04.810 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-03-09T08:58:04.810 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280284338092, "job": 1, "event": "recovery_started", "log_files": [3]} 2021-03-09T08:58:04.810 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2021-03-09T08:58:04.810 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-09T08:58:04.810 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280284338794, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1562, "table_properties": {"data_size": 648, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1615280284, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-09T08:58:04.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2021-03-09T08:58:04.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.334+0000 7fa56c31d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280284339398, "job": 1, "event": "recovery_finished"} 2021-03-09T08:58:04.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.338+0000 7fa56c31d700 4 rocksdb: DB pointer 0x564338d13800 2021-03-09T08:58:04.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.338+0000 7fa55b32d700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-09T08:58:04.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.338+0000 7fa55b32d700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-09T08:58:04.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: ** DB Stats ** 2021-03-09T08:58:04.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T08:58:04.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-09T08:58:04.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-09T08:58:04.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T08:58:04.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-09T08:58:04.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-09T08:58:04.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T08:58:04.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: ** Compaction Stats [default] ** 2021-03-09T08:58:04.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T08:58:04.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T08:58:04.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.3 0.00 0.00 1 0.001 0 0 2021-03-09T08:58:04.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.3 0.00 0.00 1 0.001 0 0 2021-03-09T08:58:04.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.3 0.00 0.00 1 0.001 0 0 2021-03-09T08:58:04.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: ** Compaction Stats [default] ** 2021-03-09T08:58:04.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T08:58:04.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T08:58:04.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.3 0.00 0.00 1 0.001 0 0 2021-03-09T08:58:04.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T08:58:04.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T08:58:04.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T08:58:04.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T08:58:04.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T08:58:04.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T08:58:04.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Cumulative compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T08:58:04.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Interval compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T08:58:04.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T08:58:04.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T08:58:04.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: ** Compaction Stats [default] ** 2021-03-09T08:58:04.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T08:58:04.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T08:58:04.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.3 0.00 0.00 1 0.001 0 0 2021-03-09T08:58:04.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.3 0.00 0.00 1 0.001 0 0 2021-03-09T08:58:04.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T08:58:04.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: ** Compaction Stats [default] ** 2021-03-09T08:58:04.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T08:58:04.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T08:58:04.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.3 0.00 0.00 1 0.001 0 0 2021-03-09T08:58:04.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T08:58:04.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T08:58:04.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T08:58:04.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T08:58:04.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T08:58:04.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T08:58:04.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Cumulative compaction: 0.00 GB write, 0.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T08:58:04.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T08:58:04.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T08:58:04.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T08:58:04.822 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.338+0000 7fa56c31d700 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2021-03-09T08:58:04.822 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.338+0000 7fa56c31d700 0 using public_addr v2:172.21.15.146:0/0 -> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] 2021-03-09T08:58:04.822 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.338+0000 7fa56c31d700 0 starting mon.b rank -1 at public addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] at bind addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:58:04.822 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.342+0000 7fa56c31d700 1 mon.b@-1(???) e0 preinit fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:58:04.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.370+0000 7fa55db32700 0 mon.b@-1(synchronizing).mds e1 new map 2021-03-09T08:58:04.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.370+0000 7fa55db32700 0 mon.b@-1(synchronizing).mds e1 print_map 2021-03-09T08:58:04.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: e1 2021-03-09T08:58:04.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: enable_multiple, ever_enabled_multiple: 1,1 2021-03-09T08:58:04.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-03-09T08:58:04.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: legacy client fscid: -1 2021-03-09T08:58:04.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: 2021-03-09T08:58:04.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: No filesystems configured 2021-03-09T08:58:04.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2021-03-09T08:58:04.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2021-03-09T08:58:04.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2021-03-09T08:58:04.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2021-03-09T08:58:04.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2021-03-09T08:58:04.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2021-03-09T08:58:04.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2021-03-09T08:58:04.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T08:58:04.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T08:58:04.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T08:58:04.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: cluster 2021-03-09T08:58:01.290090+0000 mgr.y (mgr.14144) 42 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:04.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: audit 2021-03-09T08:58:02.792770+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:04.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: audit 2021-03-09T08:58:02.797694+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:04.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 1 mon.b@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2021-03-09T08:58:04.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 20 expand_channel_meta expand map: {default=false} 2021-03-09T08:58:04.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 20 expand_channel_meta from 'false' to 'false' 2021-03-09T08:58:04.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 20 expand_channel_meta expanded map: {default=false} 2021-03-09T08:58:04.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 20 expand_channel_meta expand map: {default=info} 2021-03-09T08:58:04.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 20 expand_channel_meta from 'info' to 'info' 2021-03-09T08:58:04.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 20 expand_channel_meta expanded map: {default=info} 2021-03-09T08:58:04.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 20 expand_channel_meta expand map: {default=daemon} 2021-03-09T08:58:04.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-03-09T08:58:04.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 20 expand_channel_meta expanded map: {default=daemon} 2021-03-09T08:58:04.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 20 expand_channel_meta expand map: {default=debug} 2021-03-09T08:58:04.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 20 expand_channel_meta from 'debug' to 'debug' 2021-03-09T08:58:04.830 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 20 expand_channel_meta expanded map: {default=debug} 2021-03-09T08:58:04.830 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:04 smithi146 bash[13136]: debug 2021-03-09T08:58:04.374+0000 7fa55db32700 10 mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2021-03-09T08:58:09.868 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: audit 2021-03-09T08:58:04.391099+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-03-09T08:58:09.868 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: cluster 2021-03-09T08:58:04.391751+0000 mon.a (mon.0) 133 : cluster [INF] mon.a calling monitor election 2021-03-09T08:58:09.869 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: cluster 2021-03-09T08:58:04.392095+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2021-03-09T08:58:09.869 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: audit 2021-03-09T08:58:04.394121+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:09.869 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: audit 2021-03-09T08:58:04.394439+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-09T08:58:09.869 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: cluster 2021-03-09T08:58:05.291088+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:09.870 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: audit 2021-03-09T08:58:05.384177+0000 mon.a (mon.0) 136 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:09.870 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: audit 2021-03-09T08:58:06.384556+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:09.870 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: cluster 2021-03-09T08:58:07.291516+0000 mgr.y (mgr.14144) 45 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:09.870 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: audit 2021-03-09T08:58:07.384085+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:09.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: audit 2021-03-09T08:58:08.384551+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:09.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: audit 2021-03-09T08:58:09.384418+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:09.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: cluster 2021-03-09T08:58:09.394111+0000 mon.a (mon.0) 141 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-03-09T08:58:09.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: cluster 2021-03-09T08:58:09.399466+0000 mon.a (mon.0) 142 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2021-03-09T08:58:09.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: cluster 2021-03-09T08:58:09.399588+0000 mon.a (mon.0) 143 : cluster [DBG] fsmap 2021-03-09T08:58:09.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: cluster 2021-03-09T08:58:09.399664+0000 mon.a (mon.0) 144 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-09T08:58:09.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: cluster 2021-03-09T08:58:09.400032+0000 mon.a (mon.0) 145 : cluster [DBG] mgrmap e13: y(active, since 84s) 2021-03-09T08:58:09.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: cluster 2021-03-09T08:58:09.400338+0000 mon.a (mon.0) 146 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2021-03-09T08:58:09.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: cluster 2021-03-09T08:58:09.406652+0000 mon.a (mon.0) 147 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2021-03-09T08:58:09.873 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: cluster 2021-03-09T08:58:09.406690+0000 mon.a (mon.0) 148 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2021-03-09T08:58:09.873 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: cluster 2021-03-09T08:58:09.406749+0000 mon.a (mon.0) 149 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] is down (out of quorum) 2021-03-09T08:58:09.873 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: audit 2021-03-09T08:58:09.410967+0000 mon.a (mon.0) 150 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:09.873 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[12738]: audit 2021-03-09T08:58:09.417443+0000 mon.a (mon.0) 151 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:09.874 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: audit 2021-03-09T08:58:04.391099+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-03-09T08:58:09.874 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: cluster 2021-03-09T08:58:04.391751+0000 mon.a (mon.0) 133 : cluster [INF] mon.a calling monitor election 2021-03-09T08:58:09.874 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: cluster 2021-03-09T08:58:04.392095+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2021-03-09T08:58:09.874 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: audit 2021-03-09T08:58:04.394121+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:09.875 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: audit 2021-03-09T08:58:04.394439+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-09T08:58:09.875 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: cluster 2021-03-09T08:58:05.291088+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:09.875 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: audit 2021-03-09T08:58:05.384177+0000 mon.a (mon.0) 136 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:09.875 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: audit 2021-03-09T08:58:06.384556+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:09.875 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: cluster 2021-03-09T08:58:07.291516+0000 mgr.y (mgr.14144) 45 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:09.876 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: audit 2021-03-09T08:58:07.384085+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:09.876 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: audit 2021-03-09T08:58:08.384551+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:09.876 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: audit 2021-03-09T08:58:09.384418+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:09.877 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: cluster 2021-03-09T08:58:09.394111+0000 mon.a (mon.0) 141 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-03-09T08:58:09.877 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: cluster 2021-03-09T08:58:09.399466+0000 mon.a (mon.0) 142 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2021-03-09T08:58:09.877 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: cluster 2021-03-09T08:58:09.399588+0000 mon.a (mon.0) 143 : cluster [DBG] fsmap 2021-03-09T08:58:09.877 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: cluster 2021-03-09T08:58:09.399664+0000 mon.a (mon.0) 144 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-09T08:58:09.877 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: cluster 2021-03-09T08:58:09.400032+0000 mon.a (mon.0) 145 : cluster [DBG] mgrmap e13: y(active, since 84s) 2021-03-09T08:58:09.878 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: cluster 2021-03-09T08:58:09.400338+0000 mon.a (mon.0) 146 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2021-03-09T08:58:09.878 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: cluster 2021-03-09T08:58:09.406652+0000 mon.a (mon.0) 147 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2021-03-09T08:58:09.878 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: cluster 2021-03-09T08:58:09.406690+0000 mon.a (mon.0) 148 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2021-03-09T08:58:09.879 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: cluster 2021-03-09T08:58:09.406749+0000 mon.a (mon.0) 149 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] is down (out of quorum) 2021-03-09T08:58:09.879 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: audit 2021-03-09T08:58:09.410967+0000 mon.a (mon.0) 150 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:09.879 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:09 smithi005 bash[16589]: audit 2021-03-09T08:58:09.417443+0000 mon.a (mon.0) 151 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:10.531 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.526+0000 7fb14a5db700 1 -- 172.21.15.146:0/2994101270 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb130003250 msgr2=0x7fb1300056c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:10.531 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.526+0000 7fb14a5db700 1 --2- 172.21.15.146:0/2994101270 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb130003250 0x7fb1300056c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fb14406a8a0 tx=0x7fb13400b040).stop 2021-03-09T08:58:10.532 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.526+0000 7fb14a5db700 1 -- 172.21.15.146:0/2994101270 shutdown_connections 2021-03-09T08:58:10.532 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.526+0000 7fb14a5db700 1 --2- 172.21.15.146:0/2994101270 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb130003250 0x7fb1300056c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:10.532 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.526+0000 7fb14a5db700 1 -- 172.21.15.146:0/2994101270 >> 172.21.15.146:0/2994101270 conn(0x7fb1440f1270 msgr2=0x7fb1440f36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:58:10.532 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.526+0000 7fb14a5db700 1 -- 172.21.15.146:0/2994101270 shutdown_connections 2021-03-09T08:58:10.533 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.526+0000 7fb14a5db700 1 -- 172.21.15.146:0/2994101270 wait complete. 2021-03-09T08:58:10.534 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.526+0000 7fb14a5db700 1 Processor -- start 2021-03-09T08:58:10.534 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.526+0000 7fb14a5db700 1 -- start start 2021-03-09T08:58:10.534 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.526+0000 7fb14a5db700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb130003250 0x7fb1441072d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:10.535 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb14a5db700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1441077c0 0x7fb1441086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:10.535 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb14a5db700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb144108ba0 0x7fb144109020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:10.535 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb14a5db700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb144107e00 con 0x7fb130003250 2021-03-09T08:58:10.535 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb14a5db700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14410f900 con 0x7fb1441077c0 2021-03-09T08:58:10.536 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb14a5db700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14410fa40 con 0x7fb144108ba0 2021-03-09T08:58:10.536 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb148dd8700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1441077c0 0x7fb1441086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:10.536 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb1495d9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb130003250 0x7fb1441072d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:10.536 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb148dd8700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1441077c0 0x7fb1441086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51338/0 (socket says 172.21.15.146:51338) 2021-03-09T08:58:10.536 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb148dd8700 1 -- 172.21.15.146:0/3250583547 learned_addr learned my addr 172.21.15.146:0/3250583547 (peer_addr_for_me v2:172.21.15.146:0/0) 2021-03-09T08:58:10.537 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb149dda700 1 --2- 172.21.15.146:0/3250583547 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb144108ba0 0x7fb144109020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:10.537 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb148dd8700 1 -- 172.21.15.146:0/3250583547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1441077c0 msgr2=0x7fb1441086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T08:58:10.537 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb148dd8700 1 -- 172.21.15.146:0/3250583547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1441077c0 msgr2=0x7fb1441086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:10.537 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb148dd8700 1 --2- 172.21.15.146:0/3250583547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1441077c0 0x7fb1441086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:10.537 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb148dd8700 1 --2- 172.21.15.146:0/3250583547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1441077c0 0x7fb1441086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:10.538 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb1495d9700 1 -- 172.21.15.146:0/3250583547 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb144108ba0 msgr2=0x7fb144109020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:10.538 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb1495d9700 1 --2- 172.21.15.146:0/3250583547 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb144108ba0 0x7fb144109020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:10.538 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb1495d9700 1 -- 172.21.15.146:0/3250583547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1441077c0 msgr2=0x7fb1441086b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:10.539 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb1495d9700 1 --2- 172.21.15.146:0/3250583547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1441077c0 0x7fb1441086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:10.539 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb1495d9700 1 -- 172.21.15.146:0/3250583547 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb14410fc00 con 0x7fb130003250 2021-03-09T08:58:10.540 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb1495d9700 1 --2- 172.21.15.146:0/3250583547 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb130003250 0x7fb1441072d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fb134016960 tx=0x7fb13400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:10.541 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb139ffb700 1 -- 172.21.15.146:0/3250583547 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1340155d0 con 0x7fb130003250 2021-03-09T08:58:10.541 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb139ffb700 1 -- 172.21.15.146:0/3250583547 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7fb134015bb0 con 0x7fb130003250 2021-03-09T08:58:10.541 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb14a5db700 1 -- 172.21.15.146:0/3250583547 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb14410fe50 con 0x7fb130003250 2021-03-09T08:58:10.541 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb139ffb700 1 -- 172.21.15.146:0/3250583547 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb134020940 con 0x7fb130003250 2021-03-09T08:58:10.542 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.530+0000 7fb14a5db700 1 -- 172.21.15.146:0/3250583547 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb144110250 con 0x7fb130003250 2021-03-09T08:58:10.542 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.534+0000 7fb139ffb700 1 -- 172.21.15.146:0/3250583547 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 13) v1 ==== 36113+0+0 (secure 0 0 0) 0x7fb134030070 con 0x7fb130003250 2021-03-09T08:58:10.542 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.534+0000 7fb14a5db700 1 -- 172.21.15.146:0/3250583547 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1440589f0 con 0x7fb130003250 2021-03-09T08:58:10.542 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.534+0000 7fb139ffb700 1 --2- 172.21.15.146:0/3250583547 >> v2:172.21.15.5:6800/4269212764 conn(0x7fb128030390 0x7fb128032850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:10.542 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.534+0000 7fb139ffb700 1 -- 172.21.15.146:0/3250583547 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7fb134042280 con 0x7fb130003250 2021-03-09T08:58:10.543 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.534+0000 7fb148dd8700 1 --2- 172.21.15.146:0/3250583547 >> v2:172.21.15.5:6800/4269212764 conn(0x7fb128030390 0x7fb128032850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:10.543 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.534+0000 7fb148dd8700 1 --2- 172.21.15.146:0/3250583547 >> v2:172.21.15.5:6800/4269212764 conn(0x7fb128030390 0x7fb128032850 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb140006cf0 tx=0x7fb140008040).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:10.549 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.546+0000 7fb139ffb700 1 -- 172.21.15.146:0/3250583547 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7fb134027d70 con 0x7fb130003250 2021-03-09T08:58:10.781 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.778+0000 7fb14a5db700 1 -- 172.21.15.146:0/3250583547 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb144045280 con 0x7fb130003250 2021-03-09T08:58:10.782 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.778+0000 7fb139ffb700 1 -- 172.21.15.146:0/3250583547 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1216 (secure 0 0 0) 0x7fb13401b020 con 0x7fb130003250 2021-03-09T08:58:10.783 INFO:teuthology.orchestra.run.smithi146.stdout: 2021-03-09T08:58:10.783 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":3,"fsid":"252abbe0-80b5-11eb-907b-001a4aab830c","modified":"2021-03-09T08:58:04.384829Z","created":"2021-03-09T08:55:39.986925Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:3300","nonce":0},{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2021-03-09T08:58:10.784 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.778+0000 7fb14a5db700 1 -- 172.21.15.146:0/3250583547 >> v2:172.21.15.5:6800/4269212764 conn(0x7fb128030390 msgr2=0x7fb128032850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:10.785 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.778+0000 7fb14a5db700 1 --2- 172.21.15.146:0/3250583547 >> v2:172.21.15.5:6800/4269212764 conn(0x7fb128030390 0x7fb128032850 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb140006cf0 tx=0x7fb140008040).stop 2021-03-09T08:58:10.785 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.778+0000 7fb14a5db700 1 -- 172.21.15.146:0/3250583547 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb130003250 msgr2=0x7fb1441072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:10.785 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.778+0000 7fb14a5db700 1 --2- 172.21.15.146:0/3250583547 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb130003250 0x7fb1441072d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fb134016960 tx=0x7fb13400b040).stop 2021-03-09T08:58:10.785 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.778+0000 7fb14a5db700 1 -- 172.21.15.146:0/3250583547 shutdown_connections 2021-03-09T08:58:10.786 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.778+0000 7fb14a5db700 1 --2- 172.21.15.146:0/3250583547 >> v2:172.21.15.5:6800/4269212764 conn(0x7fb128030390 0x7fb128032850 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:10.786 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.778+0000 7fb14a5db700 1 --2- 172.21.15.146:0/3250583547 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb144108ba0 0x7fb144109020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:10.786 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.778+0000 7fb14a5db700 1 --2- 172.21.15.146:0/3250583547 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb130003250 0x7fb1441072d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:10.786 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.778+0000 7fb14a5db700 1 --2- 172.21.15.146:0/3250583547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1441077c0 0x7fb1441086b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:10.787 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.778+0000 7fb14a5db700 1 -- 172.21.15.146:0/3250583547 >> 172.21.15.146:0/3250583547 conn(0x7fb1440f1270 msgr2=0x7fb1440fbb50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:58:10.787 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.778+0000 7fb14a5db700 1 -- 172.21.15.146:0/3250583547 shutdown_connections 2021-03-09T08:58:10.787 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:10.782+0000 7fb14a5db700 1 -- 172.21.15.146:0/3250583547 wait complete. 2021-03-09T08:58:10.788 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 3 2021-03-09T08:58:10.867 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:10 smithi005 bash[12738]: cluster 2021-03-09T08:58:09.291972+0000 mgr.y (mgr.14144) 46 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:10.868 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:10 smithi005 bash[12738]: audit 2021-03-09T08:58:10.384713+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:10.868 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:10 smithi005 bash[16589]: cluster 2021-03-09T08:58:09.291972+0000 mgr.y (mgr.14144) 46 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:10.869 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:10 smithi005 bash[16589]: audit 2021-03-09T08:58:10.384713+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:11.515 INFO:tasks.cephadm:Generating final ceph.conf file... 2021-03-09T08:58:11.516 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph config generate-minimal-conf 2021-03-09T08:58:11.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[12738]: cluster 2021-03-09T08:58:06.385743+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2021-03-09T08:58:11.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[12738]: cluster 2021-03-09T08:58:11.388475+0000 mon.b (mon.2) 2 : cluster [INF] mon.b calling monitor election 2021-03-09T08:58:11.724 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[12738]: cluster 2021-03-09T08:58:11.390532+0000 mon.c (mon.1) 3 : cluster [INF] mon.c calling monitor election 2021-03-09T08:58:11.724 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[12738]: cluster 2021-03-09T08:58:11.390707+0000 mon.a (mon.0) 155 : cluster [INF] mon.a calling monitor election 2021-03-09T08:58:11.724 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[12738]: cluster 2021-03-09T08:58:11.410754+0000 mon.a (mon.0) 156 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2021-03-09T08:58:11.724 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[12738]: cluster 2021-03-09T08:58:11.417399+0000 mon.a (mon.0) 157 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2021-03-09T08:58:11.724 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[12738]: cluster 2021-03-09T08:58:11.417568+0000 mon.a (mon.0) 158 : cluster [DBG] fsmap 2021-03-09T08:58:11.725 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[12738]: cluster 2021-03-09T08:58:11.417635+0000 mon.a (mon.0) 159 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-09T08:58:11.725 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[12738]: cluster 2021-03-09T08:58:11.417975+0000 mon.a (mon.0) 160 : cluster [DBG] mgrmap e13: y(active, since 86s) 2021-03-09T08:58:11.725 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[12738]: cluster 2021-03-09T08:58:11.418296+0000 mon.a (mon.0) 161 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2021-03-09T08:58:11.725 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[12738]: cluster 2021-03-09T08:58:11.418349+0000 mon.a (mon.0) 162 : cluster [INF] Cluster is now healthy 2021-03-09T08:58:11.726 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[12738]: cluster 2021-03-09T08:58:11.425904+0000 mon.a (mon.0) 163 : cluster [INF] overall HEALTH_OK 2021-03-09T08:58:11.726 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[16589]: cluster 2021-03-09T08:58:06.385743+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2021-03-09T08:58:11.726 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[16589]: cluster 2021-03-09T08:58:11.388475+0000 mon.b (mon.2) 2 : cluster [INF] mon.b calling monitor election 2021-03-09T08:58:11.726 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[16589]: cluster 2021-03-09T08:58:11.390532+0000 mon.c (mon.1) 3 : cluster [INF] mon.c calling monitor election 2021-03-09T08:58:11.727 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[16589]: cluster 2021-03-09T08:58:11.390707+0000 mon.a (mon.0) 155 : cluster [INF] mon.a calling monitor election 2021-03-09T08:58:11.727 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[16589]: cluster 2021-03-09T08:58:11.410754+0000 mon.a (mon.0) 156 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2021-03-09T08:58:11.727 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[16589]: cluster 2021-03-09T08:58:11.417399+0000 mon.a (mon.0) 157 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2021-03-09T08:58:11.727 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[16589]: cluster 2021-03-09T08:58:11.417568+0000 mon.a (mon.0) 158 : cluster [DBG] fsmap 2021-03-09T08:58:11.727 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[16589]: cluster 2021-03-09T08:58:11.417635+0000 mon.a (mon.0) 159 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-09T08:58:11.728 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[16589]: cluster 2021-03-09T08:58:11.417975+0000 mon.a (mon.0) 160 : cluster [DBG] mgrmap e13: y(active, since 86s) 2021-03-09T08:58:11.728 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[16589]: cluster 2021-03-09T08:58:11.418296+0000 mon.a (mon.0) 161 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2021-03-09T08:58:11.728 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[16589]: cluster 2021-03-09T08:58:11.418349+0000 mon.a (mon.0) 162 : cluster [INF] Cluster is now healthy 2021-03-09T08:58:11.728 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:11 smithi005 bash[16589]: cluster 2021-03-09T08:58:11.425904+0000 mon.a (mon.0) 163 : cluster [INF] overall HEALTH_OK 2021-03-09T08:58:11.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:11 smithi146 bash[13136]: cluster 2021-03-09T08:58:09.291972+0000 mgr.y (mgr.14144) 46 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:11.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:11 smithi146 bash[13136]: audit 2021-03-09T08:58:10.384713+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:11.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:11 smithi146 bash[13136]: cluster 2021-03-09T08:58:06.385743+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2021-03-09T08:58:11.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:11 smithi146 bash[13136]: cluster 2021-03-09T08:58:11.388475+0000 mon.b (mon.2) 2 : cluster [INF] mon.b calling monitor election 2021-03-09T08:58:11.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:11 smithi146 bash[13136]: cluster 2021-03-09T08:58:11.390532+0000 mon.c (mon.1) 3 : cluster [INF] mon.c calling monitor election 2021-03-09T08:58:11.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:11 smithi146 bash[13136]: cluster 2021-03-09T08:58:11.390707+0000 mon.a (mon.0) 155 : cluster [INF] mon.a calling monitor election 2021-03-09T08:58:11.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:11 smithi146 bash[13136]: cluster 2021-03-09T08:58:11.410754+0000 mon.a (mon.0) 156 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2021-03-09T08:58:11.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:11 smithi146 bash[13136]: cluster 2021-03-09T08:58:11.417399+0000 mon.a (mon.0) 157 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2021-03-09T08:58:11.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:11 smithi146 bash[13136]: cluster 2021-03-09T08:58:11.417568+0000 mon.a (mon.0) 158 : cluster [DBG] fsmap 2021-03-09T08:58:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:11 smithi146 bash[13136]: cluster 2021-03-09T08:58:11.417635+0000 mon.a (mon.0) 159 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-09T08:58:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:11 smithi146 bash[13136]: cluster 2021-03-09T08:58:11.417975+0000 mon.a (mon.0) 160 : cluster [DBG] mgrmap e13: y(active, since 86s) 2021-03-09T08:58:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:11 smithi146 bash[13136]: cluster 2021-03-09T08:58:11.418296+0000 mon.a (mon.0) 161 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2021-03-09T08:58:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:11 smithi146 bash[13136]: cluster 2021-03-09T08:58:11.418349+0000 mon.a (mon.0) 162 : cluster [INF] Cluster is now healthy 2021-03-09T08:58:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:11 smithi146 bash[13136]: cluster 2021-03-09T08:58:11.425904+0000 mon.a (mon.0) 163 : cluster [INF] overall HEALTH_OK 2021-03-09T08:58:13.000 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:12 smithi005 bash[12738]: cluster 2021-03-09T08:58:11.292364+0000 mgr.y (mgr.14144) 47 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:13.001 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:12 smithi005 bash[12738]: audit 2021-03-09T08:58:12.385252+0000 mon.a (mon.0) 164 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:13.001 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:12 smithi005 bash[16589]: cluster 2021-03-09T08:58:11.292364+0000 mgr.y (mgr.14144) 47 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:13.001 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:12 smithi005 bash[16589]: audit 2021-03-09T08:58:12.385252+0000 mon.a (mon.0) 164 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:13.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:12 smithi146 bash[13136]: cluster 2021-03-09T08:58:11.292364+0000 mgr.y (mgr.14144) 47 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:13.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:12 smithi146 bash[13136]: audit 2021-03-09T08:58:12.385252+0000 mon.a (mon.0) 164 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-09T08:58:14.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa540e4f700 1 -- 172.21.15.5:0/3072797069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa53c05fb10 msgr2=0x7fa53c0fab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:14.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa540e4f700 1 --2- 172.21.15.5:0/3072797069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa53c05fb10 0x7fa53c0fab70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fa52c009cc0 tx=0x7fa52c004d10).stop 2021-03-09T08:58:14.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa540e4f700 1 -- 172.21.15.5:0/3072797069 shutdown_connections 2021-03-09T08:58:14.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa540e4f700 1 --2- 172.21.15.5:0/3072797069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa53c05fb10 0x7fa53c0fab70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:14.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa540e4f700 1 -- 172.21.15.5:0/3072797069 >> 172.21.15.5:0/3072797069 conn(0x7fa53c0f0170 msgr2=0x7fa53c0f25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:58:14.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa540e4f700 1 -- 172.21.15.5:0/3072797069 shutdown_connections 2021-03-09T08:58:14.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa540e4f700 1 -- 172.21.15.5:0/3072797069 wait complete. 2021-03-09T08:58:14.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa540e4f700 1 Processor -- start 2021-03-09T08:58:14.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa540e4f700 1 -- start start 2021-03-09T08:58:14.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa540e4f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa53c05fb10 0x7fa53c107400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:14.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa540e4f700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa53c1078f0 0x7fa53c1087e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:14.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa540e4f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c108cd0 0x7fa53c109150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:14.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa540e4f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c107f30 con 0x7fa53c05fb10 2021-03-09T08:58:14.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa540e4f700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c10fa30 con 0x7fa53c1078f0 2021-03-09T08:58:14.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa540e4f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c10fb70 con 0x7fa53c108cd0 2021-03-09T08:58:14.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa53bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c108cd0 0x7fa53c109150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:14.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa53b7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa53c05fb10 0x7fa53c107400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:14.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa53b7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa53c05fb10 0x7fa53c107400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46550/0 (socket says 172.21.15.5:46550) 2021-03-09T08:58:14.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa53b7fe700 1 -- 172.21.15.5:0/2303093526 learned_addr learned my addr 172.21.15.5:0/2303093526 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:58:14.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa53bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c108cd0 0x7fa53c109150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:52966/0 (socket says 172.21.15.5:52966) 2021-03-09T08:58:14.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa53affd700 1 --2- 172.21.15.5:0/2303093526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa53c1078f0 0x7fa53c1087e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:14.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa53bfff700 1 -- 172.21.15.5:0/2303093526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa53c1078f0 msgr2=0x7fa53c1087e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:14.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa53bfff700 1 --2- 172.21.15.5:0/2303093526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa53c1078f0 0x7fa53c1087e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:14.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa53bfff700 1 -- 172.21.15.5:0/2303093526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa53c05fb10 msgr2=0x7fa53c107400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:14.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa53bfff700 1 --2- 172.21.15.5:0/2303093526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa53c05fb10 0x7fa53c107400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:14.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa53bfff700 1 -- 172.21.15.5:0/2303093526 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53c10fd30 con 0x7fa53c108cd0 2021-03-09T08:58:14.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.747+0000 7fa53bfff700 1 --2- 172.21.15.5:0/2303093526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c108cd0 0x7fa53c109150 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fa534009710 tx=0x7fa534006d60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:14.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.751+0000 7fa533fff700 1 -- 172.21.15.5:0/2303093526 <== mon.1 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa534012410 con 0x7fa53c108cd0 2021-03-09T08:58:14.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.751+0000 7fa533fff700 1 -- 172.21.15.5:0/2303093526 <== mon.1 v2:172.21.15.5:3301/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7fa5340129f0 con 0x7fa53c108cd0 2021-03-09T08:58:14.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.751+0000 7fa540e4f700 1 -- 172.21.15.5:0/2303093526 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53c10ffe0 con 0x7fa53c108cd0 2021-03-09T08:58:14.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.751+0000 7fa533fff700 1 -- 172.21.15.5:0/2303093526 <== mon.1 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53400d920 con 0x7fa53c108cd0 2021-03-09T08:58:14.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.751+0000 7fa540e4f700 1 -- 172.21.15.5:0/2303093526 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53c110470 con 0x7fa53c108cd0 2021-03-09T08:58:14.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.751+0000 7fa533fff700 1 -- 172.21.15.5:0/2303093526 <== mon.1 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 13) v1 ==== 36113+0+0 (secure 0 0 0) 0x7fa53400f030 con 0x7fa53c108cd0 2021-03-09T08:58:14.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.751+0000 7fa540e4f700 1 -- 172.21.15.5:0/2303093526 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa53c0589f0 con 0x7fa53c108cd0 2021-03-09T08:58:14.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.751+0000 7fa533fff700 1 --2- 172.21.15.5:0/2303093526 >> v2:172.21.15.5:6800/4269212764 conn(0x7fa528030390 0x7fa528032850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:14.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.751+0000 7fa533fff700 1 -- 172.21.15.5:0/2303093526 <== mon.1 v2:172.21.15.5:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7fa53403f8e0 con 0x7fa53c108cd0 2021-03-09T08:58:14.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.751+0000 7fa53b7fe700 1 --2- 172.21.15.5:0/2303093526 >> v2:172.21.15.5:6800/4269212764 conn(0x7fa528030390 0x7fa528032850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:14.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.751+0000 7fa53b7fe700 1 --2- 172.21.15.5:0/2303093526 >> v2:172.21.15.5:6800/4269212764 conn(0x7fa528030390 0x7fa528032850 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fa52c005010 tx=0x7fa52c004d10).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:14.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.759+0000 7fa533fff700 1 -- 172.21.15.5:0/2303093526 <== mon.1 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7fa534012570 con 0x7fa53c108cd0 2021-03-09T08:58:14.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.931+0000 7fa540e4f700 1 -- 172.21.15.5:0/2303093526 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa53c045280 con 0x7fa53c108cd0 2021-03-09T08:58:14.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.931+0000 7fa533fff700 1 -- 172.21.15.5:0/2303093526 <== mon.1 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v11) v1 ==== 76+0+269 (secure 0 0 0) 0x7fa534010360 con 0x7fa53c108cd0 2021-03-09T08:58:14.933 INFO:teuthology.orchestra.run.smithi005.stdout:# minimal ceph.conf for 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:58:14.934 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2021-03-09T08:58:14.934 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T08:58:14.934 INFO:teuthology.orchestra.run.smithi005.stdout: mon_host = [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2021-03-09T08:58:14.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.935+0000 7fa540e4f700 1 -- 172.21.15.5:0/2303093526 >> v2:172.21.15.5:6800/4269212764 conn(0x7fa528030390 msgr2=0x7fa528032850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:14.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.935+0000 7fa540e4f700 1 --2- 172.21.15.5:0/2303093526 >> v2:172.21.15.5:6800/4269212764 conn(0x7fa528030390 0x7fa528032850 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fa52c005010 tx=0x7fa52c004d10).stop 2021-03-09T08:58:14.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.935+0000 7fa540e4f700 1 -- 172.21.15.5:0/2303093526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c108cd0 msgr2=0x7fa53c109150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:14.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.935+0000 7fa540e4f700 1 --2- 172.21.15.5:0/2303093526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c108cd0 0x7fa53c109150 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fa534009710 tx=0x7fa534006d60).stop 2021-03-09T08:58:14.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.935+0000 7fa540e4f700 1 -- 172.21.15.5:0/2303093526 shutdown_connections 2021-03-09T08:58:14.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.935+0000 7fa540e4f700 1 --2- 172.21.15.5:0/2303093526 >> v2:172.21.15.5:6800/4269212764 conn(0x7fa528030390 0x7fa528032850 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:14.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.935+0000 7fa540e4f700 1 --2- 172.21.15.5:0/2303093526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa53c108cd0 0x7fa53c109150 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:14.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.935+0000 7fa540e4f700 1 --2- 172.21.15.5:0/2303093526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa53c05fb10 0x7fa53c107400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:14.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.935+0000 7fa540e4f700 1 --2- 172.21.15.5:0/2303093526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa53c1078f0 0x7fa53c1087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:14.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.935+0000 7fa540e4f700 1 -- 172.21.15.5:0/2303093526 >> 172.21.15.5:0/2303093526 conn(0x7fa53c0f0170 msgr2=0x7fa53c0f0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:58:14.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.935+0000 7fa540e4f700 1 -- 172.21.15.5:0/2303093526 shutdown_connections 2021-03-09T08:58:14.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:14.935+0000 7fa540e4f700 1 -- 172.21.15.5:0/2303093526 wait complete. 2021-03-09T08:58:15.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:14 smithi005 bash[12738]: cluster 2021-03-09T08:58:13.292944+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:15.143 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:14 smithi005 bash[16589]: cluster 2021-03-09T08:58:13.292944+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:15.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:14 smithi146 bash[13136]: cluster 2021-03-09T08:58:13.292944+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:15.689 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2021-03-09T08:58:15.690 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2021-03-09T08:58:15.691 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.conf 2021-03-09T08:58:15.703 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2021-03-09T08:58:15.704 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-03-09T08:58:15.763 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2021-03-09T08:58:15.764 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.conf 2021-03-09T08:58:15.780 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2021-03-09T08:58:15.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-03-09T08:58:15.839 INFO:tasks.cephadm:Adding mgr.y on smithi005 2021-03-09T08:58:15.840 INFO:tasks.cephadm:Adding mgr.x on smithi146 2021-03-09T08:58:15.840 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph orch apply mgr '3;smithi005=y;smithi146=x' 2021-03-09T08:58:16.035 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:15 smithi146 bash[13136]: audit 2021-03-09T08:58:14.933167+0000 mon.c (mon.1) 4 : audit [DBG] from='client.? 172.21.15.5:0/2303093526' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:16.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:15 smithi005 bash[12738]: audit 2021-03-09T08:58:14.933167+0000 mon.c (mon.1) 4 : audit [DBG] from='client.? 172.21.15.5:0/2303093526' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:16.118 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:15 smithi005 bash[16589]: audit 2021-03-09T08:58:14.933167+0000 mon.c (mon.1) 4 : audit [DBG] from='client.? 172.21.15.5:0/2303093526' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:17.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:16 smithi146 bash[13136]: cluster 2021-03-09T08:58:15.293472+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:17.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:16 smithi005 bash[12738]: cluster 2021-03-09T08:58:15.293472+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:17.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:17 smithi005 bash[16589]: cluster 2021-03-09T08:58:15.293472+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:18.722 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.714+0000 7f60f3fff700 1 -- 172.21.15.146:0/119649044 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60f40f6280 msgr2=0x7f60f40fcb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:18.723 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.714+0000 7f60f3fff700 1 --2- 172.21.15.146:0/119649044 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60f40f6280 0x7f60f40fcb80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f60ec004e30 tx=0x7f60ec009b30).stop 2021-03-09T08:58:18.723 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.714+0000 7f60f3fff700 1 -- 172.21.15.146:0/119649044 shutdown_connections 2021-03-09T08:58:18.723 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.714+0000 7f60f3fff700 1 --2- 172.21.15.146:0/119649044 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60f40f6280 0x7f60f40fcb80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:18.724 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.714+0000 7f60f3fff700 1 --2- 172.21.15.146:0/119649044 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60f40f4690 0x7f60f40f4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:18.724 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.714+0000 7f60f3fff700 1 --2- 172.21.15.146:0/119649044 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60f40f5930 0x7f60f40f5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:18.724 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.714+0000 7f60f3fff700 1 -- 172.21.15.146:0/119649044 >> 172.21.15.146:0/119649044 conn(0x7f60f40efe90 msgr2=0x7f60f40f22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:58:18.724 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f3fff700 1 -- 172.21.15.146:0/119649044 shutdown_connections 2021-03-09T08:58:18.725 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f3fff700 1 -- 172.21.15.146:0/119649044 wait complete. 2021-03-09T08:58:18.725 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f3fff700 1 Processor -- start 2021-03-09T08:58:18.725 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f3fff700 1 -- start start 2021-03-09T08:58:18.725 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60f40f4690 0x7f60f41080c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:18.725 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f3fff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60f40f5930 0x7f60f41085b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:18.726 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60f40f6280 0x7f60f41094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:18.726 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f3fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f41099d0 con 0x7f60f40f6280 2021-03-09T08:58:18.726 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f3fff700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f4109b40 con 0x7f60f40f5930 2021-03-09T08:58:18.726 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f4109d00 con 0x7f60f40f4690 2021-03-09T08:58:18.726 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f27fc700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60f40f5930 0x7f60f41085b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:18.727 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f27fc700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60f40f5930 0x7f60f41085b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51358/0 (socket says 172.21.15.146:51358) 2021-03-09T08:58:18.727 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f2ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60f40f4690 0x7f60f41080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:18.727 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f27fc700 1 -- 172.21.15.146:0/4098076933 learned_addr learned my addr 172.21.15.146:0/4098076933 (peer_addr_for_me v2:172.21.15.146:0/0) 2021-03-09T08:58:18.727 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f37fe700 1 --2- 172.21.15.146:0/4098076933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60f40f6280 0x7f60f41094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:18.728 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f27fc700 1 -- 172.21.15.146:0/4098076933 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60f40f4690 msgr2=0x7f60f41080c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:18.728 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f27fc700 1 --2- 172.21.15.146:0/4098076933 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60f40f4690 0x7f60f41080c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:18.728 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f27fc700 1 -- 172.21.15.146:0/4098076933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60f40f6280 msgr2=0x7f60f41094a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:18.728 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f27fc700 1 --2- 172.21.15.146:0/4098076933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60f40f6280 0x7f60f41094a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:18.728 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f27fc700 1 -- 172.21.15.146:0/4098076933 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f4109ec0 con 0x7f60f40f5930 2021-03-09T08:58:18.729 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f27fc700 1 --2- 172.21.15.146:0/4098076933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60f40f5930 0x7f60f41085b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f60dc00eec0 tx=0x7f60dc00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:18.729 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60eb7fe700 1 -- 172.21.15.146:0/4098076933 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc0097b0 con 0x7f60f40f5930 2021-03-09T08:58:18.729 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60eb7fe700 1 -- 172.21.15.146:0/4098076933 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f60dc00f040 con 0x7f60f40f5930 2021-03-09T08:58:18.731 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f3fff700 1 -- 172.21.15.146:0/4098076933 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f410a170 con 0x7f60f40f5930 2021-03-09T08:58:18.731 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60eb7fe700 1 -- 172.21.15.146:0/4098076933 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc007ad0 con 0x7f60f40f5930 2021-03-09T08:58:18.732 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60f3fff700 1 -- 172.21.15.146:0/4098076933 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f4110900 con 0x7f60f40f5930 2021-03-09T08:58:18.732 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60eb7fe700 1 -- 172.21.15.146:0/4098076933 <== mon.2 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) v1 ==== 36113+0+0 (secure 0 0 0) 0x7f60dc01b030 con 0x7f60f40f5930 2021-03-09T08:58:18.732 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60eb7fe700 1 --2- 172.21.15.146:0/4098076933 >> v2:172.21.15.5:6800/4269212764 conn(0x7f60e00301a0 0x7f60e0032660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:18.732 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.718+0000 7f60eb7fe700 1 -- 172.21.15.146:0/4098076933 <== mon.2 v2:172.21.15.146:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f60dc040900 con 0x7f60f40f5930 2021-03-09T08:58:18.733 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.722+0000 7f60f2ffd700 1 --2- 172.21.15.146:0/4098076933 >> v2:172.21.15.5:6800/4269212764 conn(0x7f60e00301a0 0x7f60e0032660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:18.733 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.722+0000 7f60f3fff700 1 -- 172.21.15.146:0/4098076933 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60f40f3a80 con 0x7f60f40f5930 2021-03-09T08:58:18.733 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.722+0000 7f60f2ffd700 1 --2- 172.21.15.146:0/4098076933 >> v2:172.21.15.5:6800/4269212764 conn(0x7f60e00301a0 0x7f60e0032660 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f60e4009fd0 tx=0x7f60e400b040).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:18.733 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.726+0000 7f60eb7fe700 1 -- 172.21.15.146:0/4098076933 <== mon.2 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f60dc019070 con 0x7f60f40f5930 2021-03-09T08:58:18.921 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.918+0000 7f60f3fff700 1 -- 172.21.15.146:0/4098076933 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi005=y;smithi146=x", "target": ["mon-mgr", ""]}) v1 -- 0x7f60f40275c0 con 0x7f60e00301a0 2021-03-09T08:58:18.932 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.926+0000 7f60eb7fe700 1 -- 172.21.15.146:0/4098076933 <== mgr.14144 v2:172.21.15.5:6800/4269212764 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f60f40275c0 con 0x7f60e00301a0 2021-03-09T08:58:18.932 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled mgr update... 2021-03-09T08:58:18.935 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.930+0000 7f60f3fff700 1 -- 172.21.15.146:0/4098076933 >> v2:172.21.15.5:6800/4269212764 conn(0x7f60e00301a0 msgr2=0x7f60e0032660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:18.935 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.930+0000 7f60f3fff700 1 --2- 172.21.15.146:0/4098076933 >> v2:172.21.15.5:6800/4269212764 conn(0x7f60e00301a0 0x7f60e0032660 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f60e4009fd0 tx=0x7f60e400b040).stop 2021-03-09T08:58:18.935 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.930+0000 7f60f3fff700 1 -- 172.21.15.146:0/4098076933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60f40f5930 msgr2=0x7f60f41085b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:18.936 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.930+0000 7f60f3fff700 1 --2- 172.21.15.146:0/4098076933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60f40f5930 0x7f60f41085b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f60dc00eec0 tx=0x7f60dc00b040).stop 2021-03-09T08:58:18.936 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.930+0000 7f60f3fff700 1 -- 172.21.15.146:0/4098076933 shutdown_connections 2021-03-09T08:58:18.937 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.930+0000 7f60f3fff700 1 --2- 172.21.15.146:0/4098076933 >> v2:172.21.15.5:6800/4269212764 conn(0x7f60e00301a0 0x7f60e0032660 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:18.937 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.930+0000 7f60f3fff700 1 --2- 172.21.15.146:0/4098076933 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60f40f6280 0x7f60f41094a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:18.937 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.930+0000 7f60f3fff700 1 --2- 172.21.15.146:0/4098076933 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f60f40f4690 0x7f60f41080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:18.937 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.930+0000 7f60f3fff700 1 --2- 172.21.15.146:0/4098076933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60f40f5930 0x7f60f41085b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:18.938 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.930+0000 7f60f3fff700 1 -- 172.21.15.146:0/4098076933 >> 172.21.15.146:0/4098076933 conn(0x7f60f40efe90 msgr2=0x7f60f40f1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:58:18.938 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.930+0000 7f60f3fff700 1 -- 172.21.15.146:0/4098076933 shutdown_connections 2021-03-09T08:58:18.938 INFO:teuthology.orchestra.run.smithi146.stderr:2021-03-09T08:58:18.930+0000 7f60f3fff700 1 -- 172.21.15.146:0/4098076933 wait complete. 2021-03-09T08:58:19.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:18 smithi146 bash[13136]: cluster 2021-03-09T08:58:17.293940+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:19.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:18 smithi005 bash[12738]: cluster 2021-03-09T08:58:17.293940+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:19.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:18 smithi005 bash[16589]: cluster 2021-03-09T08:58:17.293940+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:19.570 DEBUG:teuthology.orchestra.run.smithi146:mgr.x> sudo journalctl -f -n 0 -u ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mgr.x.service 2021-03-09T08:58:19.573 INFO:tasks.cephadm:Deploying OSDs... 2021-03-09T08:58:19.573 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2021-03-09T08:58:19.574 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2021-03-09T08:58:19.581 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-09T08:58:19.582 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-09T08:58:19.589 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-09T08:58:19.589 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:58:19.590 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 357 Links: 1 2021-03-09T08:58:19.590 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:58:19.590 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2021-03-09 08:52:25.536537260 +0000 2021-03-09T08:58:19.590 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2021-03-09 08:52:25.352537264 +0000 2021-03-09T08:58:19.590 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2021-03-09 08:52:25.352537264 +0000 2021-03-09T08:58:19.591 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2021-03-09T08:58:19.597 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2021-03-09T08:58:19.598 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2021-03-09T08:58:19.598 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.00029736 s, 1.7 MB/s 2021-03-09T08:58:19.602 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-09T08:58:19.605 INFO:journalctl@ceph.mgr.x.smithi146.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-09T08:58:19.649 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-09T08:58:19.650 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:58:19.650 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 363 Links: 1 2021-03-09T08:58:19.650 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:58:19.651 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2021-03-09 08:52:25.752537255 +0000 2021-03-09T08:58:19.651 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2021-03-09 08:52:25.748537255 +0000 2021-03-09T08:58:19.651 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2021-03-09 08:52:25.748537255 +0000 2021-03-09T08:58:19.652 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2021-03-09T08:58:19.660 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2021-03-09T08:58:19.660 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2021-03-09T08:58:19.661 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000306741 s, 1.7 MB/s 2021-03-09T08:58:19.666 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-09T08:58:19.673 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-09T08:58:19.673 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:58:19.674 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 367 Links: 1 2021-03-09T08:58:19.674 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:58:19.674 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2021-03-09 08:52:26.368537241 +0000 2021-03-09T08:58:19.674 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2021-03-09 08:52:26.180537245 +0000 2021-03-09T08:58:19.675 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2021-03-09 08:52:26.180537245 +0000 2021-03-09T08:58:19.675 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2021-03-09T08:58:19.684 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2021-03-09T08:58:19.684 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2021-03-09T08:58:19.684 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000294848 s, 1.7 MB/s 2021-03-09T08:58:19.689 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-09T08:58:19.696 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-09T08:58:19.696 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:58:19.697 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 373 Links: 1 2021-03-09T08:58:19.697 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:58:19.697 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2021-03-09 08:52:26.576537236 +0000 2021-03-09T08:58:19.697 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2021-03-09 08:52:26.572537237 +0000 2021-03-09T08:58:19.697 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2021-03-09 08:52:26.572537237 +0000 2021-03-09T08:58:19.698 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2021-03-09T08:58:19.706 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2021-03-09T08:58:19.706 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2021-03-09T08:58:19.706 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000293906 s, 1.7 MB/s 2021-03-09T08:58:19.711 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2021-03-09T08:58:19.711 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2021-03-09T08:58:19.718 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-09T08:58:19.718 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-09T08:58:19.766 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-09T08:58:19.766 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:58:19.767 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 346 Links: 1 2021-03-09T08:58:19.767 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:58:19.767 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2021-03-09 08:52:33.185915884 +0000 2021-03-09T08:58:19.767 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2021-03-09 08:52:32.997915886 +0000 2021-03-09T08:58:19.768 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2021-03-09 08:52:32.997915886 +0000 2021-03-09T08:58:19.768 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2021-03-09T08:58:19.775 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2021-03-09T08:58:19.775 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2021-03-09T08:58:19.775 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000313184 s, 1.6 MB/s 2021-03-09T08:58:19.778 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-09T08:58:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-09T08:58:19.784 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:58:19.784 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 352 Links: 1 2021-03-09T08:58:19.784 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:58:19.785 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2021-03-09 08:52:33.601915880 +0000 2021-03-09T08:58:19.785 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2021-03-09 08:52:33.437915882 +0000 2021-03-09T08:58:19.785 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2021-03-09 08:52:33.437915882 +0000 2021-03-09T08:58:19.785 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2021-03-09T08:58:19.788 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2021-03-09T08:58:19.789 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2021-03-09T08:58:19.789 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000221399 s, 2.3 MB/s 2021-03-09T08:58:19.792 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-09T08:58:19.797 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-09T08:58:19.798 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:58:19.798 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 358 Links: 1 2021-03-09T08:58:19.798 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:58:19.799 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2021-03-09 08:52:34.037915875 +0000 2021-03-09T08:58:19.799 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2021-03-09 08:52:33.849915877 +0000 2021-03-09T08:58:19.799 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2021-03-09 08:52:33.849915877 +0000 2021-03-09T08:58:19.799 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2021-03-09T08:58:19.804 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2021-03-09T08:58:19.804 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2021-03-09T08:58:19.804 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.00032254 s, 1.6 MB/s 2021-03-09T08:58:19.808 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-09T08:58:19.858 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-09T08:58:19.859 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:58:19.859 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 366 Links: 1 2021-03-09T08:58:19.859 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:58:19.859 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2021-03-09 08:52:34.421915871 +0000 2021-03-09T08:58:19.860 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2021-03-09 08:52:34.249915873 +0000 2021-03-09T08:58:19.860 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2021-03-09 08:52:34.249915873 +0000 2021-03-09T08:58:19.860 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2021-03-09T08:58:19.868 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2021-03-09T08:58:19.869 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2021-03-09T08:58:19.869 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000411447 s, 1.2 MB/s 2021-03-09T08:58:19.874 INFO:tasks.cephadm:Deploying osd.0 on smithi005 with /dev/vg_nvme/lv_4... 2021-03-09T08:58:19.874 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-03-09T08:58:20.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:19 smithi146 bash[13136]: audit 2021-03-09T08:58:18.923716+0000 mgr.y (mgr.14144) 51 : audit [DBG] from='client.34106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi005=y;smithi146=x", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:58:20.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:19 smithi146 bash[13136]: cephadm 2021-03-09T08:58:18.925414+0000 mgr.y (mgr.14144) 52 : cephadm [INF] Saving service mgr spec with placement smithi005=y;smithi146=x;count:3 2021-03-09T08:58:20.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:19 smithi146 bash[13136]: audit 2021-03-09T08:58:18.931376+0000 mon.a (mon.0) 165 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:20.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:19 smithi146 bash[13136]: audit 2021-03-09T08:58:18.932656+0000 mon.a (mon.0) 166 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:20.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:19 smithi146 bash[13136]: cephadm 2021-03-09T08:58:18.934753+0000 mgr.y (mgr.14144) 53 : cephadm [INF] Refreshing smithi005 facts 2021-03-09T08:58:20.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:19 smithi146 bash[13136]: audit 2021-03-09T08:58:19.385520+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:20.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:19 smithi146 bash[13136]: audit 2021-03-09T08:58:19.387142+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-09T08:58:20.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:19 smithi146 bash[13136]: audit 2021-03-09T08:58:19.393006+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-03-09T08:58:20.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:19 smithi146 bash[13136]: audit 2021-03-09T08:58:19.394294+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-09T08:58:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:19 smithi146 bash[13136]: audit 2021-03-09T08:58:19.395667+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:20.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[12738]: audit 2021-03-09T08:58:18.923716+0000 mgr.y (mgr.14144) 51 : audit [DBG] from='client.34106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi005=y;smithi146=x", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:58:20.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[12738]: cephadm 2021-03-09T08:58:18.925414+0000 mgr.y (mgr.14144) 52 : cephadm [INF] Saving service mgr spec with placement smithi005=y;smithi146=x;count:3 2021-03-09T08:58:20.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[12738]: audit 2021-03-09T08:58:18.931376+0000 mon.a (mon.0) 165 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:20.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[12738]: audit 2021-03-09T08:58:18.932656+0000 mon.a (mon.0) 166 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:20.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[12738]: cephadm 2021-03-09T08:58:18.934753+0000 mgr.y (mgr.14144) 53 : cephadm [INF] Refreshing smithi005 facts 2021-03-09T08:58:20.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[12738]: audit 2021-03-09T08:58:19.385520+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:20.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[12738]: audit 2021-03-09T08:58:19.387142+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-09T08:58:20.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[12738]: audit 2021-03-09T08:58:19.393006+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-03-09T08:58:20.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[12738]: audit 2021-03-09T08:58:19.394294+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-09T08:58:20.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[12738]: audit 2021-03-09T08:58:19.395667+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:20.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[16589]: audit 2021-03-09T08:58:18.923716+0000 mgr.y (mgr.14144) 51 : audit [DBG] from='client.34106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi005=y;smithi146=x", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:58:20.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[16589]: cephadm 2021-03-09T08:58:18.925414+0000 mgr.y (mgr.14144) 52 : cephadm [INF] Saving service mgr spec with placement smithi005=y;smithi146=x;count:3 2021-03-09T08:58:20.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[16589]: audit 2021-03-09T08:58:18.931376+0000 mon.a (mon.0) 165 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:20.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[16589]: audit 2021-03-09T08:58:18.932656+0000 mon.a (mon.0) 166 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:20.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[16589]: cephadm 2021-03-09T08:58:18.934753+0000 mgr.y (mgr.14144) 53 : cephadm [INF] Refreshing smithi005 facts 2021-03-09T08:58:20.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[16589]: audit 2021-03-09T08:58:19.385520+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:20.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[16589]: audit 2021-03-09T08:58:19.387142+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-09T08:58:20.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[16589]: audit 2021-03-09T08:58:19.393006+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-03-09T08:58:20.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[16589]: audit 2021-03-09T08:58:19.394294+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-09T08:58:20.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:19 smithi005 bash[16589]: audit 2021-03-09T08:58:19.395667+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:21.141 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:20 smithi146 bash[13136]: cluster 2021-03-09T08:58:19.294445+0000 mgr.y (mgr.14144) 54 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:21.141 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:20 smithi146 bash[13136]: cephadm 2021-03-09T08:58:19.396843+0000 mgr.y (mgr.14144) 55 : cephadm [INF] Deploying daemon mgr.x on smithi146 2021-03-09T08:58:21.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:20 smithi005 bash[12738]: cluster 2021-03-09T08:58:19.294445+0000 mgr.y (mgr.14144) 54 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:21.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:20 smithi005 bash[12738]: cephadm 2021-03-09T08:58:19.396843+0000 mgr.y (mgr.14144) 55 : cephadm [INF] Deploying daemon mgr.x on smithi146 2021-03-09T08:58:21.283 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:20 smithi005 bash[16589]: cluster 2021-03-09T08:58:19.294445+0000 mgr.y (mgr.14144) 54 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:21.284 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:20 smithi005 bash[16589]: cephadm 2021-03-09T08:58:19.396843+0000 mgr.y (mgr.14144) 55 : cephadm [INF] Deploying daemon mgr.x on smithi146 2021-03-09T08:58:23.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:22 smithi005 bash[12738]: cluster 2021-03-09T08:58:21.294916+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:23.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:22 smithi005 bash[12738]: audit 2021-03-09T08:58:22.148096+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:23.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:22 smithi005 bash[12738]: audit 2021-03-09T08:58:22.149740+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:23.118 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:22 smithi005 bash[16589]: cluster 2021-03-09T08:58:21.294916+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:23.118 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:22 smithi005 bash[16589]: audit 2021-03-09T08:58:22.148096+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:23.118 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:22 smithi005 bash[16589]: audit 2021-03-09T08:58:22.149740+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:23.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:22 smithi146 bash[13136]: cluster 2021-03-09T08:58:21.294916+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:23.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:22 smithi146 bash[13136]: audit 2021-03-09T08:58:22.148096+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:23.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:22 smithi146 bash[13136]: audit 2021-03-09T08:58:22.149740+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:23.478 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-03-09T08:58:23.478 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-03-09T08:58:23.478 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10+0 records in 2021-03-09T08:58:23.479 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2021-03-09T08:58:23.479 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110317 s, 951 MB/s 2021-03-09T08:58:23.479 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping successful for: 2021-03-09T08:58:23.512 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Mar 09 08:58:23 smithi146 bash[13804]: debug 2021-03-09T08:58:23.222+0000 7f5befda0700 1 -- 172.21.15.146:0/3102180795 <== mon.2 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5614bb5924e0 con 0x5614bb682c00 2021-03-09T08:58:24.188 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph orch daemon add osd smithi005:vg_nvme/lv_4 2021-03-09T08:58:25.259 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:24 smithi005 bash[12738]: cluster 2021-03-09T08:58:23.295366+0000 mgr.y (mgr.14144) 57 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:25.259 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:24 smithi005 bash[16589]: cluster 2021-03-09T08:58:23.295366+0000 mgr.y (mgr.14144) 57 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:24 smithi146 bash[13136]: cluster 2021-03-09T08:58:23.295366+0000 mgr.y (mgr.14144) 57 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:27.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:26 smithi146 bash[13136]: cluster 2021-03-09T08:58:25.295811+0000 mgr.y (mgr.14144) 58 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:27.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:26 smithi146 bash[13136]: audit 2021-03-09T08:58:25.632295+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:27.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:26 smithi146 bash[13136]: audit 2021-03-09T08:58:25.637977+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:27.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:26 smithi146 bash[13136]: cephadm 2021-03-09T08:58:25.639445+0000 mgr.y (mgr.14144) 59 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2021-03-09T08:58:27.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:26 smithi146 bash[13136]: audit 2021-03-09T08:58:25.640115+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-09T08:58:27.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:26 smithi146 bash[13136]: audit 2021-03-09T08:58:25.642018+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-09T08:58:27.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:26 smithi146 bash[13136]: audit 2021-03-09T08:58:25.643662+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:27.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:26 smithi146 bash[13136]: cephadm 2021-03-09T08:58:25.645019+0000 mgr.y (mgr.14144) 60 : cephadm [INF] Reconfiguring daemon mgr.y on smithi005 2021-03-09T08:58:27.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:26 smithi005 bash[12738]: cluster 2021-03-09T08:58:25.295811+0000 mgr.y (mgr.14144) 58 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:27.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:26 smithi005 bash[12738]: audit 2021-03-09T08:58:25.632295+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:27.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:26 smithi005 bash[12738]: audit 2021-03-09T08:58:25.637977+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:27.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:26 smithi005 bash[12738]: cephadm 2021-03-09T08:58:25.639445+0000 mgr.y (mgr.14144) 59 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2021-03-09T08:58:27.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:26 smithi005 bash[12738]: audit 2021-03-09T08:58:25.640115+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-09T08:58:27.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:26 smithi005 bash[12738]: audit 2021-03-09T08:58:25.642018+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-09T08:58:27.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:26 smithi005 bash[12738]: audit 2021-03-09T08:58:25.643662+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:27.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:26 smithi005 bash[12738]: cephadm 2021-03-09T08:58:25.645019+0000 mgr.y (mgr.14144) 60 : cephadm [INF] Reconfiguring daemon mgr.y on smithi005 2021-03-09T08:58:27.119 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:26 smithi005 bash[16589]: cluster 2021-03-09T08:58:25.295811+0000 mgr.y (mgr.14144) 58 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:27.120 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:26 smithi005 bash[16589]: audit 2021-03-09T08:58:25.632295+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:27.120 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:26 smithi005 bash[16589]: audit 2021-03-09T08:58:25.637977+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:27.120 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:26 smithi005 bash[16589]: cephadm 2021-03-09T08:58:25.639445+0000 mgr.y (mgr.14144) 59 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2021-03-09T08:58:27.120 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:26 smithi005 bash[16589]: audit 2021-03-09T08:58:25.640115+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-09T08:58:27.120 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:26 smithi005 bash[16589]: audit 2021-03-09T08:58:25.642018+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-09T08:58:27.121 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:26 smithi005 bash[16589]: audit 2021-03-09T08:58:25.643662+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:27.121 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:26 smithi005 bash[16589]: cephadm 2021-03-09T08:58:25.645019+0000 mgr.y (mgr.14144) 60 : cephadm [INF] Reconfiguring daemon mgr.y on smithi005 2021-03-09T08:58:27.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 -- 172.21.15.5:0/3196514780 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9cf00f9680 msgr2=0x7f9cf00f9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:27.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 --2- 172.21.15.5:0/3196514780 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9cf00f9680 0x7f9cf00f9ae0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9ce4004e30 tx=0x7f9ce4009b30).stop 2021-03-09T08:58:27.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 -- 172.21.15.5:0/3196514780 shutdown_connections 2021-03-09T08:58:27.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 --2- 172.21.15.5:0/3196514780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cf00f9fd0 0x7f9cf01008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:27.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 --2- 172.21.15.5:0/3196514780 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9cf00f83e0 0x7f9cf00f8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:27.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 --2- 172.21.15.5:0/3196514780 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9cf00f9680 0x7f9cf00f9ae0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:27.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 -- 172.21.15.5:0/3196514780 >> 172.21.15.5:0/3196514780 conn(0x7f9cf00f3bc0 msgr2=0x7f9cf00f6000 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:58:27.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 -- 172.21.15.5:0/3196514780 shutdown_connections 2021-03-09T08:58:27.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 -- 172.21.15.5:0/3196514780 wait complete. 2021-03-09T08:58:27.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 Processor -- start 2021-03-09T08:58:27.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 -- start start 2021-03-09T08:58:27.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9cf00f83e0 0x7f9cf01083f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:27.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9cf00f9680 0x7f9cf01088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:27.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cf00f9fd0 0x7f9cf010f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:27.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf010feb0 con 0x7f9cf00f9680 2021-03-09T08:58:27.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf010fff0 con 0x7f9cf00f9fd0 2021-03-09T08:58:27.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cf5680700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf0110130 con 0x7f9cf00f83e0 2021-03-09T08:58:27.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cee7fc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9cf00f9680 0x7f9cf01088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:27.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cee7fc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9cf00f9680 0x7f9cf01088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46566/0 (socket says 172.21.15.5:46566) 2021-03-09T08:58:27.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.775+0000 7f9cee7fc700 1 -- 172.21.15.5:0/4195857706 learned_addr learned my addr 172.21.15.5:0/4195857706 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:58:27.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9ceeffd700 1 --2- 172.21.15.5:0/4195857706 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9cf00f83e0 0x7f9cf01083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:27.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9cef7fe700 1 --2- 172.21.15.5:0/4195857706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cf00f9fd0 0x7f9cf010f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:27.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9cee7fc700 1 -- 172.21.15.5:0/4195857706 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9cf00f83e0 msgr2=0x7f9cf01083f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:27.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9cee7fc700 1 --2- 172.21.15.5:0/4195857706 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9cf00f83e0 0x7f9cf01083f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:27.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9cee7fc700 1 -- 172.21.15.5:0/4195857706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cf00f9fd0 msgr2=0x7f9cf010f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:27.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9cee7fc700 1 --2- 172.21.15.5:0/4195857706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cf00f9fd0 0x7f9cf010f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:27.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9cee7fc700 1 -- 172.21.15.5:0/4195857706 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf01102f0 con 0x7f9cf00f9680 2021-03-09T08:58:27.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9cee7fc700 1 --2- 172.21.15.5:0/4195857706 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9cf00f9680 0x7f9cf01088e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f9ce4004ca0 tx=0x7f9ce4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:27.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9cd77fe700 1 -- 172.21.15.5:0/4195857706 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce400e030 con 0x7f9cf00f9680 2021-03-09T08:58:27.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9cd77fe700 1 -- 172.21.15.5:0/4195857706 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f9ce4013aa0 con 0x7f9cf00f9680 2021-03-09T08:58:27.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9cf5680700 1 -- 172.21.15.5:0/4195857706 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf0110540 con 0x7f9cf00f9680 2021-03-09T08:58:27.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9cd77fe700 1 -- 172.21.15.5:0/4195857706 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4018070 con 0x7f9cf00f9680 2021-03-09T08:58:27.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9cf5680700 1 -- 172.21.15.5:0/4195857706 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf0110940 con 0x7f9cf00f9680 2021-03-09T08:58:27.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9cd77fe700 1 -- 172.21.15.5:0/4195857706 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 13) v1 ==== 36113+0+0 (secure 0 0 0) 0x7f9ce401daa0 con 0x7f9cf00f9680 2021-03-09T08:58:27.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9cd77fe700 1 --2- 172.21.15.5:0/4195857706 >> v2:172.21.15.5:6800/4269212764 conn(0x7f9cd80300e0 0x7f9cd80325a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:27.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9cd77fe700 1 -- 172.21.15.5:0/4195857706 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f9ce4041a60 con 0x7f9cf00f9680 2021-03-09T08:58:27.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9ceeffd700 1 --2- 172.21.15.5:0/4195857706 >> v2:172.21.15.5:6800/4269212764 conn(0x7f9cd80300e0 0x7f9cd80325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:27.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9cf5680700 1 -- 172.21.15.5:0/4195857706 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf0045ae0 con 0x7f9cf00f9680 2021-03-09T08:58:27.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.779+0000 7f9ceeffd700 1 --2- 172.21.15.5:0/4195857706 >> v2:172.21.15.5:6800/4269212764 conn(0x7f9cd80300e0 0x7f9cd80325a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9cf00f94e0 tx=0x7f9ce000b040).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:27.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.787+0000 7f9cd77fe700 1 -- 172.21.15.5:0/4195857706 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f9ce40263e0 con 0x7f9cf00f9680 2021-03-09T08:58:27.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:27.979+0000 7f9cf5680700 1 -- 172.21.15.5:0/4195857706 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cf0103240 con 0x7f9cd80300e0 2021-03-09T08:58:29.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:28 smithi005 bash[12738]: cluster 2021-03-09T08:58:27.296251+0000 mgr.y (mgr.14144) 61 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:29.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:28 smithi005 bash[12738]: audit 2021-03-09T08:58:27.985380+0000 mgr.y (mgr.14144) 62 : audit [DBG] from='client.14202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:58:29.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:28 smithi005 bash[12738]: audit 2021-03-09T08:58:27.987024+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-09T08:58:29.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:28 smithi005 bash[12738]: cephadm 2021-03-09T08:58:27.990153+0000 mgr.y (mgr.14144) 63 : cephadm [INF] Applying service osd.None on host smithi005... 2021-03-09T08:58:29.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:28 smithi005 bash[12738]: audit 2021-03-09T08:58:27.990662+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-09T08:58:29.194 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:28 smithi005 bash[12738]: audit 2021-03-09T08:58:27.992072+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:29.194 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:28 smithi005 bash[12738]: audit 2021-03-09T08:58:28.371705+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:29.194 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:28 smithi005 bash[12738]: audit 2021-03-09T08:58:28.372723+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:29.194 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:28 smithi005 bash[16589]: cluster 2021-03-09T08:58:27.296251+0000 mgr.y (mgr.14144) 61 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:29.195 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:28 smithi005 bash[16589]: audit 2021-03-09T08:58:27.985380+0000 mgr.y (mgr.14144) 62 : audit [DBG] from='client.14202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:58:29.195 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:28 smithi005 bash[16589]: audit 2021-03-09T08:58:27.987024+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-09T08:58:29.195 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:28 smithi005 bash[16589]: cephadm 2021-03-09T08:58:27.990153+0000 mgr.y (mgr.14144) 63 : cephadm [INF] Applying service osd.None on host smithi005... 2021-03-09T08:58:29.195 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:28 smithi005 bash[16589]: audit 2021-03-09T08:58:27.990662+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-09T08:58:29.196 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:28 smithi005 bash[16589]: audit 2021-03-09T08:58:27.992072+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:29.196 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:28 smithi005 bash[16589]: audit 2021-03-09T08:58:28.371705+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:29.196 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:28 smithi005 bash[16589]: audit 2021-03-09T08:58:28.372723+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:29.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:28 smithi146 bash[13136]: cluster 2021-03-09T08:58:27.296251+0000 mgr.y (mgr.14144) 61 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:29.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:28 smithi146 bash[13136]: audit 2021-03-09T08:58:27.985380+0000 mgr.y (mgr.14144) 62 : audit [DBG] from='client.14202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:58:29.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:28 smithi146 bash[13136]: audit 2021-03-09T08:58:27.987024+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-09T08:58:29.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:28 smithi146 bash[13136]: cephadm 2021-03-09T08:58:27.990153+0000 mgr.y (mgr.14144) 63 : cephadm [INF] Applying service osd.None on host smithi005... 2021-03-09T08:58:29.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:28 smithi146 bash[13136]: audit 2021-03-09T08:58:27.990662+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-09T08:58:29.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:28 smithi146 bash[13136]: audit 2021-03-09T08:58:27.992072+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:28 smithi146 bash[13136]: audit 2021-03-09T08:58:28.371705+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:28 smithi146 bash[13136]: audit 2021-03-09T08:58:28.372723+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:29.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:29.891+0000 7f9cd77fe700 1 -- 172.21.15.5:0/4195857706 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f9ce4027ce0 con 0x7f9cf00f9680 2021-03-09T08:58:30.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:29 smithi146 bash[13136]: cluster 2021-03-09T08:58:29.269074+0000 mon.a (mon.0) 184 : cluster [DBG] Standby manager daemon x started 2021-03-09T08:58:30.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:29 smithi146 bash[13136]: audit 2021-03-09T08:58:29.269158+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.146:0/3820672966' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-03-09T08:58:30.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:29 smithi146 bash[13136]: audit 2021-03-09T08:58:29.271564+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.146:0/3820672966' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-03-09T08:58:30.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:29 smithi005 bash[12738]: cluster 2021-03-09T08:58:29.269074+0000 mon.a (mon.0) 184 : cluster [DBG] Standby manager daemon x started 2021-03-09T08:58:30.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:29 smithi005 bash[12738]: audit 2021-03-09T08:58:29.269158+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.146:0/3820672966' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-03-09T08:58:30.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:29 smithi005 bash[12738]: audit 2021-03-09T08:58:29.271564+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.146:0/3820672966' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-03-09T08:58:30.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:29 smithi005 bash[16589]: cluster 2021-03-09T08:58:29.269074+0000 mon.a (mon.0) 184 : cluster [DBG] Standby manager daemon x started 2021-03-09T08:58:30.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:29 smithi005 bash[16589]: audit 2021-03-09T08:58:29.269158+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.146:0/3820672966' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-03-09T08:58:30.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:29 smithi005 bash[16589]: audit 2021-03-09T08:58:29.271564+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.146:0/3820672966' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-03-09T08:58:31.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:30 smithi146 bash[13136]: cluster 2021-03-09T08:58:29.296692+0000 mgr.y (mgr.14144) 64 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:31.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:30 smithi146 bash[13136]: cluster 2021-03-09T08:58:29.895992+0000 mon.a (mon.0) 185 : cluster [DBG] mgrmap e14: y(active, since 104s), standbys: x 2021-03-09T08:58:31.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:30 smithi146 bash[13136]: audit 2021-03-09T08:58:29.896555+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-03-09T08:58:31.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:30 smithi005 bash[12738]: cluster 2021-03-09T08:58:29.296692+0000 mgr.y (mgr.14144) 64 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:31.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:30 smithi005 bash[12738]: cluster 2021-03-09T08:58:29.895992+0000 mon.a (mon.0) 185 : cluster [DBG] mgrmap e14: y(active, since 104s), standbys: x 2021-03-09T08:58:31.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:30 smithi005 bash[12738]: audit 2021-03-09T08:58:29.896555+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-03-09T08:58:31.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:30 smithi005 bash[16589]: cluster 2021-03-09T08:58:29.296692+0000 mgr.y (mgr.14144) 64 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:31.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:30 smithi005 bash[16589]: cluster 2021-03-09T08:58:29.895992+0000 mon.a (mon.0) 185 : cluster [DBG] mgrmap e14: y(active, since 104s), standbys: x 2021-03-09T08:58:31.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:30 smithi005 bash[16589]: audit 2021-03-09T08:58:29.896555+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-03-09T08:58:33.117 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:32 smithi005 bash[16589]: cluster 2021-03-09T08:58:31.297165+0000 mgr.y (mgr.14144) 65 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:33.117 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:32 smithi005 bash[16589]: audit 2021-03-09T08:58:31.858488+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:33.118 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:32 smithi005 bash[16589]: audit 2021-03-09T08:58:31.865030+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:33.118 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:32 smithi005 bash[16589]: audit 2021-03-09T08:58:32.154005+0000 mon.c (mon.1) 5 : audit [INF] from='client.? 172.21.15.5:0/3013145266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98424831-48d3-4def-876b-a45c8d94dd9e"}]: dispatch 2021-03-09T08:58:33.118 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:32 smithi005 bash[16589]: audit 2021-03-09T08:58:32.154990+0000 mon.a (mon.0) 189 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98424831-48d3-4def-876b-a45c8d94dd9e"}]: dispatch 2021-03-09T08:58:33.118 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:32 smithi005 bash[16589]: audit 2021-03-09T08:58:32.162637+0000 mon.a (mon.0) 190 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98424831-48d3-4def-876b-a45c8d94dd9e"}]': finished 2021-03-09T08:58:33.118 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:32 smithi005 bash[16589]: cluster 2021-03-09T08:58:32.162775+0000 mon.a (mon.0) 191 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2021-03-09T08:58:33.119 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:32 smithi005 bash[16589]: audit 2021-03-09T08:58:32.162935+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-09T08:58:33.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:32 smithi005 bash[12738]: cluster 2021-03-09T08:58:31.297165+0000 mgr.y (mgr.14144) 65 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:33.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:32 smithi005 bash[12738]: audit 2021-03-09T08:58:31.858488+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:33.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:32 smithi005 bash[12738]: audit 2021-03-09T08:58:31.865030+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:33.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:32 smithi005 bash[12738]: audit 2021-03-09T08:58:32.154005+0000 mon.c (mon.1) 5 : audit [INF] from='client.? 172.21.15.5:0/3013145266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98424831-48d3-4def-876b-a45c8d94dd9e"}]: dispatch 2021-03-09T08:58:33.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:32 smithi005 bash[12738]: audit 2021-03-09T08:58:32.154990+0000 mon.a (mon.0) 189 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98424831-48d3-4def-876b-a45c8d94dd9e"}]: dispatch 2021-03-09T08:58:33.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:32 smithi005 bash[12738]: audit 2021-03-09T08:58:32.162637+0000 mon.a (mon.0) 190 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98424831-48d3-4def-876b-a45c8d94dd9e"}]': finished 2021-03-09T08:58:33.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:32 smithi005 bash[12738]: cluster 2021-03-09T08:58:32.162775+0000 mon.a (mon.0) 191 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2021-03-09T08:58:33.121 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:32 smithi005 bash[12738]: audit 2021-03-09T08:58:32.162935+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-09T08:58:33.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:32 smithi146 bash[13136]: cluster 2021-03-09T08:58:31.297165+0000 mgr.y (mgr.14144) 65 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:33.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:32 smithi146 bash[13136]: audit 2021-03-09T08:58:31.858488+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:33.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:32 smithi146 bash[13136]: audit 2021-03-09T08:58:31.865030+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:33.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:32 smithi146 bash[13136]: audit 2021-03-09T08:58:32.154005+0000 mon.c (mon.1) 5 : audit [INF] from='client.? 172.21.15.5:0/3013145266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98424831-48d3-4def-876b-a45c8d94dd9e"}]: dispatch 2021-03-09T08:58:33.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:32 smithi146 bash[13136]: audit 2021-03-09T08:58:32.154990+0000 mon.a (mon.0) 189 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98424831-48d3-4def-876b-a45c8d94dd9e"}]: dispatch 2021-03-09T08:58:33.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:32 smithi146 bash[13136]: audit 2021-03-09T08:58:32.162637+0000 mon.a (mon.0) 190 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98424831-48d3-4def-876b-a45c8d94dd9e"}]': finished 2021-03-09T08:58:33.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:32 smithi146 bash[13136]: cluster 2021-03-09T08:58:32.162775+0000 mon.a (mon.0) 191 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2021-03-09T08:58:33.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:32 smithi146 bash[13136]: audit 2021-03-09T08:58:32.162935+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-09T08:58:34.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:33 smithi005 bash[12738]: audit 2021-03-09T08:58:32.904005+0000 mon.a (mon.0) 193 : audit [DBG] from='client.? 172.21.15.5:0/2809481387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-09T08:58:34.117 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:33 smithi005 bash[16589]: audit 2021-03-09T08:58:32.904005+0000 mon.a (mon.0) 193 : audit [DBG] from='client.? 172.21.15.5:0/2809481387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-09T08:58:34.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:33 smithi146 bash[13136]: audit 2021-03-09T08:58:32.904005+0000 mon.a (mon.0) 193 : audit [DBG] from='client.? 172.21.15.5:0/2809481387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-09T08:58:35.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:34 smithi146 bash[13136]: cluster 2021-03-09T08:58:33.297668+0000 mgr.y (mgr.14144) 66 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:35.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:34 smithi005 bash[12738]: cluster 2021-03-09T08:58:33.297668+0000 mgr.y (mgr.14144) 66 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:35.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:34 smithi005 bash[16589]: cluster 2021-03-09T08:58:33.297668+0000 mgr.y (mgr.14144) 66 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:37.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:36 smithi005 bash[12738]: cluster 2021-03-09T08:58:35.298164+0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:37.117 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:36 smithi005 bash[16589]: cluster 2021-03-09T08:58:35.298164+0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:37.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:36 smithi146 bash[13136]: cluster 2021-03-09T08:58:35.298164+0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:39.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:38 smithi146 bash[13136]: cluster 2021-03-09T08:58:37.298686+0000 mgr.y (mgr.14144) 68 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:39.308 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:38 smithi005 bash[12738]: cluster 2021-03-09T08:58:37.298686+0000 mgr.y (mgr.14144) 68 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:39.308 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:38 smithi005 bash[16589]: cluster 2021-03-09T08:58:37.298686+0000 mgr.y (mgr.14144) 68 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:41.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:40 smithi146 bash[13136]: cluster 2021-03-09T08:58:39.299239+0000 mgr.y (mgr.14144) 69 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:41.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:40 smithi146 bash[13136]: audit 2021-03-09T08:58:40.523521+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-03-09T08:58:41.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:40 smithi146 bash[13136]: audit 2021-03-09T08:58:40.525209+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:41.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:40 smithi005 bash[12738]: cluster 2021-03-09T08:58:39.299239+0000 mgr.y (mgr.14144) 69 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:41.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:40 smithi005 bash[12738]: audit 2021-03-09T08:58:40.523521+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-03-09T08:58:41.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:40 smithi005 bash[12738]: audit 2021-03-09T08:58:40.525209+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:41.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:40 smithi005 bash[16589]: cluster 2021-03-09T08:58:39.299239+0000 mgr.y (mgr.14144) 69 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:41.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:40 smithi005 bash[16589]: audit 2021-03-09T08:58:40.523521+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-03-09T08:58:41.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:40 smithi005 bash[16589]: audit 2021-03-09T08:58:40.525209+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:42.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:41 smithi146 bash[13136]: cephadm 2021-03-09T08:58:40.526738+0000 mgr.y (mgr.14144) 70 : cephadm [INF] Deploying daemon osd.0 on smithi005 2021-03-09T08:58:42.271 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:41 smithi005 bash[12738]: cephadm 2021-03-09T08:58:40.526738+0000 mgr.y (mgr.14144) 70 : cephadm [INF] Deploying daemon osd.0 on smithi005 2021-03-09T08:58:42.272 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:41 smithi005 bash[16589]: cephadm 2021-03-09T08:58:40.526738+0000 mgr.y (mgr.14144) 70 : cephadm [INF] Deploying daemon osd.0 on smithi005 2021-03-09T08:58:43.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:42 smithi005 bash[12738]: cluster 2021-03-09T08:58:41.299722+0000 mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:43.117 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:42 smithi005 bash[16589]: cluster 2021-03-09T08:58:41.299722+0000 mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:43.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:42 smithi146 bash[13136]: cluster 2021-03-09T08:58:41.299722+0000 mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:43.295 INFO:teuthology.orchestra.run.smithi005.stdout:Created osd(s) 0 on host 'smithi005' 2021-03-09T08:58:43.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:43.291+0000 7f9cd77fe700 1 -- 172.21.15.5:0/4195857706 <== mgr.14144 v2:172.21.15.5:6800/4269212764 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9cf0103240 con 0x7f9cd80300e0 2021-03-09T08:58:43.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:43.295+0000 7f9cf5680700 1 -- 172.21.15.5:0/4195857706 >> v2:172.21.15.5:6800/4269212764 conn(0x7f9cd80300e0 msgr2=0x7f9cd80325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:43.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:43.295+0000 7f9cf5680700 1 --2- 172.21.15.5:0/4195857706 >> v2:172.21.15.5:6800/4269212764 conn(0x7f9cd80300e0 0x7f9cd80325a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9cf00f94e0 tx=0x7f9ce000b040).stop 2021-03-09T08:58:43.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:43.295+0000 7f9cf5680700 1 -- 172.21.15.5:0/4195857706 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9cf00f9680 msgr2=0x7f9cf01088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:43.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:43.295+0000 7f9cf5680700 1 --2- 172.21.15.5:0/4195857706 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9cf00f9680 0x7f9cf01088e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f9ce4004ca0 tx=0x7f9ce4009b30).stop 2021-03-09T08:58:43.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:43.295+0000 7f9cf5680700 1 -- 172.21.15.5:0/4195857706 shutdown_connections 2021-03-09T08:58:43.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:43.295+0000 7f9cf5680700 1 --2- 172.21.15.5:0/4195857706 >> v2:172.21.15.5:6800/4269212764 conn(0x7f9cd80300e0 0x7f9cd80325a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:43.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:43.295+0000 7f9cf5680700 1 --2- 172.21.15.5:0/4195857706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cf00f9fd0 0x7f9cf010f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:43.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:43.295+0000 7f9cf5680700 1 --2- 172.21.15.5:0/4195857706 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9cf00f83e0 0x7f9cf01083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:43.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:43.295+0000 7f9cf5680700 1 --2- 172.21.15.5:0/4195857706 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9cf00f9680 0x7f9cf01088e0 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0x7f9ce4004ca0 tx=0x7f9ce4009b30).stop 2021-03-09T08:58:43.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:43.295+0000 7f9cf5680700 1 -- 172.21.15.5:0/4195857706 >> 172.21.15.5:0/4195857706 conn(0x7f9cf00f3bc0 msgr2=0x7f9cf00fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:58:43.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:43.299+0000 7f9cf5680700 1 -- 172.21.15.5:0/4195857706 shutdown_connections 2021-03-09T08:58:43.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:43.299+0000 7f9cf5680700 1 -- 172.21.15.5:0/4195857706 wait complete. 2021-03-09T08:58:44.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:43 smithi146 bash[13136]: audit 2021-03-09T08:58:43.281052+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:44.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:43 smithi146 bash[13136]: audit 2021-03-09T08:58:43.287212+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:44.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:43 smithi146 bash[13136]: audit 2021-03-09T08:58:43.293349+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:44.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:43 smithi005 bash[12738]: audit 2021-03-09T08:58:43.281052+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:44.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:43 smithi005 bash[12738]: audit 2021-03-09T08:58:43.287212+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:44.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:43 smithi005 bash[12738]: audit 2021-03-09T08:58:43.293349+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:44.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:43 smithi005 bash[16589]: audit 2021-03-09T08:58:43.281052+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:44.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:43 smithi005 bash[16589]: audit 2021-03-09T08:58:43.287212+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:44.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:43 smithi005 bash[16589]: audit 2021-03-09T08:58:43.293349+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:44.685 DEBUG:teuthology.orchestra.run.smithi005:osd.0> sudo journalctl -f -n 0 -u ceph-252abbe0-80b5-11eb-907b-001a4aab830c@osd.0.service 2021-03-09T08:58:44.687 INFO:tasks.cephadm:Deploying osd.1 on smithi005 with /dev/vg_nvme/lv_3... 2021-03-09T08:58:44.688 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-03-09T08:58:44.736 INFO:journalctl@ceph.osd.0.smithi005.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-09T08:58:45.041 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:44 smithi005 bash[12738]: cephadm 2021-03-09T08:58:43.283897+0000 mgr.y (mgr.14144) 72 : cephadm [INF] Refreshing smithi146 facts 2021-03-09T08:58:45.041 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:44 smithi005 bash[12738]: cluster 2021-03-09T08:58:43.300058+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:45.042 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:44 smithi005 bash[16589]: cephadm 2021-03-09T08:58:43.283897+0000 mgr.y (mgr.14144) 72 : cephadm [INF] Refreshing smithi146 facts 2021-03-09T08:58:45.042 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:44 smithi005 bash[16589]: cluster 2021-03-09T08:58:43.300058+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:45.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:44 smithi146 bash[13136]: cephadm 2021-03-09T08:58:43.283897+0000 mgr.y (mgr.14144) 72 : cephadm [INF] Refreshing smithi146 facts 2021-03-09T08:58:45.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:44 smithi146 bash[13136]: cluster 2021-03-09T08:58:43.300058+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:46.079 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:45 smithi005 bash[12738]: audit 2021-03-09T08:58:45.483259+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-09T08:58:46.079 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:45 smithi005 bash[12738]: audit 2021-03-09T08:58:45.486298+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-09T08:58:46.080 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:45 smithi005 bash[16589]: audit 2021-03-09T08:58:45.483259+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-09T08:58:46.080 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:45 smithi005 bash[16589]: audit 2021-03-09T08:58:45.486298+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-09T08:58:46.080 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:45 smithi005 bash[18944]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-03-09T08:58:46.081 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:45 smithi005 bash[18944]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2021-03-09T08:58:46.081 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:45 smithi005 bash[18944]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2021-03-09T08:58:46.081 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:45 smithi005 bash[18944]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2021-03-09T08:58:46.081 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:45 smithi005 bash[18944]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2021-03-09T08:58:46.081 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:45 smithi005 bash[18944]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-03-09T08:58:46.082 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:45 smithi005 bash[18944]: --> ceph-volume lvm activate successful for osd ID: 0 2021-03-09T08:58:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:45 smithi146 bash[13136]: audit 2021-03-09T08:58:45.483259+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-09T08:58:46.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:45 smithi146 bash[13136]: audit 2021-03-09T08:58:45.486298+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-09T08:58:47.109 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:46 smithi005 bash[12738]: cluster 2021-03-09T08:58:45.300391+0000 mgr.y (mgr.14144) 74 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:47.109 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:46 smithi005 bash[16589]: cluster 2021-03-09T08:58:45.300391+0000 mgr.y (mgr.14144) 74 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:46 smithi146 bash[13136]: cluster 2021-03-09T08:58:45.300391+0000 mgr.y (mgr.14144) 74 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:49.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[12738]: cluster 2021-03-09T08:58:47.300790+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:49.094 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[16589]: cluster 2021-03-09T08:58:47.300790+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:49.095 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b584dd700 10 -- 172.21.15.5:0/810713003 dispatch_throttle_release 413 to dispatch throttler 413/104857600 2021-03-09T08:58:49.095 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b584dd700 20 -- 172.21.15.5:0/810713003 done calling dispatch on 0x55e1970b54a0 2021-03-09T08:58:49.095 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 10 -- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 msgr2=0x55e1970a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 160 remaining bytes 0 2021-03-09T08:58:49.095 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 10 --2- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 0x55e1970a6300 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x55e1970b9ef0 tx=0x55e19f2ecd80).write_message sending 0x55e19f2ddb00 done. 2021-03-09T08:58:49.096 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 20 -- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 msgr2=0x55e1970a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2021-03-09T08:58:49.096 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 20 -- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 msgr2=0x55e1970a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=96 2021-03-09T08:58:49.096 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 20 --2- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 0x55e1970a6300 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x55e1970b9ef0 tx=0x55e19f2ecd80).handle_read_frame_preamble_main r=0 2021-03-09T08:58:49.096 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 20 --2- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 0x55e1970a6300 secure :-1 s=THROTTLE_MESSAGE pgs=9 cs=0 l=1 rev1=1 rx=0x55e1970b9ef0 tx=0x55e19f2ecd80).throttle_message 2021-03-09T08:58:49.097 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 20 --2- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 0x55e1970a6300 secure :-1 s=THROTTLE_BYTES pgs=9 cs=0 l=1 rev1=1 rx=0x55e1970b9ef0 tx=0x55e19f2ecd80).throttle_bytes 2021-03-09T08:58:49.097 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 20 --2- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 0x55e1970a6300 secure :-1 s=THROTTLE_DISPATCH_QUEUE pgs=9 cs=0 l=1 rev1=1 rx=0x55e1970b9ef0 tx=0x55e19f2ecd80).throttle_dispatch_queue 2021-03-09T08:58:49.097 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 20 --2- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 0x55e1970a6300 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rev1=1 rx=0x55e1970b9ef0 tx=0x55e19f2ecd80).read_frame_segment seg_idx=0 2021-03-09T08:58:49.097 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 20 --2- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 0x55e1970a6300 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rev1=1 rx=0x55e1970b9ef0 tx=0x55e19f2ecd80).read_frame_segment seg_idx=1 2021-03-09T08:58:49.097 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 20 -- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 msgr2=0x55e1970a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=208 2021-03-09T08:58:49.098 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 20 --2- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 0x55e1970a6300 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rev1=1 rx=0x55e1970b9ef0 tx=0x55e19f2ecd80).handle_read_frame_segment r=0 2021-03-09T08:58:49.098 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 20 -- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 msgr2=0x55e1970a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2021-03-09T08:58:49.098 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 20 --2- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 0x55e1970a6300 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rev1=1 rx=0x55e1970b9ef0 tx=0x55e19f2ecd80).handle_read_frame_epilogue_main r=0 2021-03-09T08:58:49.098 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 10 --2- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 0x55e1970a6300 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rev1=1 rx=0x55e1970b9ef0 tx=0x55e19f2ecd80).handle_read_frame_dispatch tag=17 2021-03-09T08:58:49.099 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 20 --2- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 0x55e1970a6300 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rev1=1 rx=0x55e1970b9ef0 tx=0x55e19f2ecd80).handle_message 2021-03-09T08:58:49.099 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 5 --2- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 0x55e1970a6300 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rev1=1 rx=0x55e1970b9ef0 tx=0x55e19f2ecd80).handle_message got 194 + 0 + 0 byte message. envelope type=18 src mon.1 off 0 2021-03-09T08:58:49.099 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 5 --2- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 0x55e1970a6300 secure :-1 s=READ_MESSAGE_COMPLETE pgs=9 cs=0 l=1 rev1=1 rx=0x55e1970b9ef0 tx=0x55e19f2ecd80).handle_message received message m=0x55e19f2bc4e0 seq=4 from=mon.1 type=18 auth_reply(proto 2 0 (0) Success) v1 2021-03-09T08:58:49.099 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 20 -- 172.21.15.5:0/810713003 queue 0x55e19f2bc4e0 prio 196 2021-03-09T08:58:49.100 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 20 --2- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 0x55e1970a6300 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x55e1970b9ef0 tx=0x55e19f2ecd80).read_frame 2021-03-09T08:58:49.100 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:48 smithi005 bash[19133]: debug 2021-03-09T08:58:48.775+0000 7f4b59ce0700 20 -- 172.21.15.5:0/810713003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x55e19f339400 msgr2=0x55e1970a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=96 2021-03-09T08:58:49.100 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:49 smithi005 bash[19133]: debug 2021-03-09T08:58:49.091+0000 7f4b5ff72080 -1 Falling back to public interface 2021-03-09T08:58:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:48 smithi146 bash[13136]: cluster 2021-03-09T08:58:47.300790+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:50.048 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-03-09T08:58:50.048 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-03-09T08:58:50.048 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10+0 records in 2021-03-09T08:58:50.048 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2021-03-09T08:58:50.049 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0145343 s, 721 MB/s 2021-03-09T08:58:50.049 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping successful for: 2021-03-09T08:58:50.253 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:49 smithi005 bash[12738]: audit 2021-03-09T08:58:48.934806+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:50.254 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:49 smithi005 bash[16589]: audit 2021-03-09T08:58:48.934806+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:49 smithi146 bash[13136]: audit 2021-03-09T08:58:48.934806+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:50.822 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph orch daemon add osd smithi005:vg_nvme/lv_3 2021-03-09T08:58:51.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:50 smithi005 bash[12738]: cluster 2021-03-09T08:58:49.301123+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:51.194 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:50 smithi005 bash[16589]: cluster 2021-03-09T08:58:49.301123+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:51.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:50 smithi146 bash[13136]: cluster 2021-03-09T08:58:49.301123+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:52.366 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:52 smithi005 bash[19133]: debug 2021-03-09T08:58:52.123+0000 7f4b5ff72080 -1 osd.0 0 log_to_monitors {default=true} 2021-03-09T08:58:53.111 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:52 smithi005 bash[12738]: cluster 2021-03-09T08:58:51.301544+0000 mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:53.112 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:52 smithi005 bash[12738]: audit 2021-03-09T08:58:52.129697+0000 mon.c (mon.1) 6 : audit [INF] from='osd.0 v2:172.21.15.5:6801/2804368853' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-03-09T08:58:53.112 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:52 smithi005 bash[12738]: audit 2021-03-09T08:58:52.130317+0000 mon.a (mon.0) 202 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-03-09T08:58:53.113 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:52 smithi005 bash[16589]: cluster 2021-03-09T08:58:51.301544+0000 mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:53.113 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:52 smithi005 bash[16589]: audit 2021-03-09T08:58:52.129697+0000 mon.c (mon.1) 6 : audit [INF] from='osd.0 v2:172.21.15.5:6801/2804368853' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-03-09T08:58:53.114 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:52 smithi005 bash[16589]: audit 2021-03-09T08:58:52.130317+0000 mon.a (mon.0) 202 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-03-09T08:58:53.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:52 smithi146 bash[13136]: cluster 2021-03-09T08:58:51.301544+0000 mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:53.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:52 smithi146 bash[13136]: audit 2021-03-09T08:58:52.129697+0000 mon.c (mon.1) 6 : audit [INF] from='osd.0 v2:172.21.15.5:6801/2804368853' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-03-09T08:58:53.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:52 smithi146 bash[13136]: audit 2021-03-09T08:58:52.130317+0000 mon.a (mon.0) 202 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-03-09T08:58:54.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:53 smithi005 bash[12738]: audit 2021-03-09T08:58:52.958551+0000 mon.a (mon.0) 203 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-03-09T08:58:54.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:53 smithi005 bash[12738]: cluster 2021-03-09T08:58:52.958667+0000 mon.a (mon.0) 204 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-03-09T08:58:54.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:53 smithi005 bash[12738]: audit 2021-03-09T08:58:52.958870+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-09T08:58:54.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:53 smithi005 bash[12738]: audit 2021-03-09T08:58:52.960729+0000 mon.c (mon.1) 7 : audit [INF] from='osd.0 v2:172.21.15.5:6801/2804368853' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2021-03-09T08:58:54.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:53 smithi005 bash[12738]: audit 2021-03-09T08:58:52.961502+0000 mon.a (mon.0) 206 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2021-03-09T08:58:54.119 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:53 smithi005 bash[16589]: audit 2021-03-09T08:58:52.958551+0000 mon.a (mon.0) 203 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-03-09T08:58:54.119 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:53 smithi005 bash[16589]: cluster 2021-03-09T08:58:52.958667+0000 mon.a (mon.0) 204 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-03-09T08:58:54.119 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:53 smithi005 bash[16589]: audit 2021-03-09T08:58:52.958870+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-09T08:58:54.120 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:53 smithi005 bash[16589]: audit 2021-03-09T08:58:52.960729+0000 mon.c (mon.1) 7 : audit [INF] from='osd.0 v2:172.21.15.5:6801/2804368853' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2021-03-09T08:58:54.120 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:53 smithi005 bash[16589]: audit 2021-03-09T08:58:52.961502+0000 mon.a (mon.0) 206 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2021-03-09T08:58:54.120 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 09 08:58:53 smithi005 bash[19133]: debug 2021-03-09T08:58:53.971+0000 7f4b564d9700 -1 osd.0 0 waiting for initial osdmap 2021-03-09T08:58:54.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:53 smithi146 bash[13136]: audit 2021-03-09T08:58:52.958551+0000 mon.a (mon.0) 203 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-03-09T08:58:54.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:53 smithi146 bash[13136]: cluster 2021-03-09T08:58:52.958667+0000 mon.a (mon.0) 204 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-03-09T08:58:54.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:53 smithi146 bash[13136]: audit 2021-03-09T08:58:52.958870+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-09T08:58:54.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:53 smithi146 bash[13136]: audit 2021-03-09T08:58:52.960729+0000 mon.c (mon.1) 7 : audit [INF] from='osd.0 v2:172.21.15.5:6801/2804368853' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2021-03-09T08:58:54.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:53 smithi146 bash[13136]: audit 2021-03-09T08:58:52.961502+0000 mon.a (mon.0) 206 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2021-03-09T08:58:55.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.003+0000 7f173b3e8700 1 -- 172.21.15.5:0/1625107665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17340f7630 msgr2=0x7f17340f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:55.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.003+0000 7f173b3e8700 1 --2- 172.21.15.5:0/1625107665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17340f7630 0x7f17340f7a50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f1730004e00 tx=0x7f1730009b30).stop 2021-03-09T08:58:55.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.003+0000 7f173b3e8700 1 -- 172.21.15.5:0/1625107665 shutdown_connections 2021-03-09T08:58:55.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.003+0000 7f173b3e8700 1 --2- 172.21.15.5:0/1625107665 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17340f46d0 0x7f17340f4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:55.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.003+0000 7f173b3e8700 1 --2- 172.21.15.5:0/1625107665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17340f7630 0x7f17340f7a50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:55.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.003+0000 7f173b3e8700 1 --2- 172.21.15.5:0/1625107665 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17340f7fd0 0x7f17340f4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:55.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.003+0000 7f173b3e8700 1 -- 172.21.15.5:0/1625107665 >> 172.21.15.5:0/1625107665 conn(0x7f17340efe90 msgr2=0x7f17340f22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:58:55.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.003+0000 7f173b3e8700 1 -- 172.21.15.5:0/1625107665 shutdown_connections 2021-03-09T08:58:55.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.003+0000 7f173b3e8700 1 -- 172.21.15.5:0/1625107665 wait complete. 2021-03-09T08:58:55.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173b3e8700 1 Processor -- start 2021-03-09T08:58:55.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173b3e8700 1 -- start start 2021-03-09T08:58:55.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173b3e8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17340f46d0 0x7f17340f6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:55.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173b3e8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17340f7630 0x7f17340f5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:55.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173b3e8700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17340f7fd0 0x7f17340f59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:55.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173b3e8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17340f60c0 con 0x7f17340f46d0 2021-03-09T08:58:55.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173b3e8700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17340f6230 con 0x7f17340f7fd0 2021-03-09T08:58:55.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173b3e8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17341100c0 con 0x7f17340f7630 2021-03-09T08:58:55.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173a3e6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17340f46d0 0x7f17340f6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:55.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173a3e6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17340f46d0 0x7f17340f6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46628/0 (socket says 172.21.15.5:46628) 2021-03-09T08:58:55.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173a3e6700 1 -- 172.21.15.5:0/1547583324 learned_addr learned my addr 172.21.15.5:0/1547583324 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:58:55.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f1739be5700 1 --2- 172.21.15.5:0/1547583324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17340f7630 0x7f17340f5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:55.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173abe7700 1 --2- 172.21.15.5:0/1547583324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17340f7fd0 0x7f17340f59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:55.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173a3e6700 1 -- 172.21.15.5:0/1547583324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17340f7630 msgr2=0x7f17340f5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:55.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173a3e6700 1 --2- 172.21.15.5:0/1547583324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17340f7630 0x7f17340f5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:55.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173a3e6700 1 -- 172.21.15.5:0/1547583324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17340f7fd0 msgr2=0x7f17340f59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:58:55.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173a3e6700 1 --2- 172.21.15.5:0/1547583324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17340f7fd0 0x7f17340f59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:55.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173a3e6700 1 -- 172.21.15.5:0/1547583324 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1734110280 con 0x7f17340f46d0 2021-03-09T08:58:55.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173abe7700 1 --2- 172.21.15.5:0/1547583324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17340f7fd0 0x7f17340f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-03-09T08:58:55.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173a3e6700 1 --2- 172.21.15.5:0/1547583324 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17340f46d0 0x7f17340f6f00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f173000b690 tx=0x7f1730009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:55.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f172affd700 1 -- 172.21.15.5:0/1547583324 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173000e030 con 0x7f17340f46d0 2021-03-09T08:58:55.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173b3e8700 1 -- 172.21.15.5:0/1547583324 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17341104d0 con 0x7f17340f46d0 2021-03-09T08:58:55.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173b3e8700 1 -- 172.21.15.5:0/1547583324 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17341109e0 con 0x7f17340f46d0 2021-03-09T08:58:55.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f172affd700 1 -- 172.21.15.5:0/1547583324 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f17300041a0 con 0x7f17340f46d0 2021-03-09T08:58:55.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f172affd700 1 -- 172.21.15.5:0/1547583324 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1730018070 con 0x7f17340f46d0 2021-03-09T08:58:55.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.007+0000 7f173b3e8700 1 -- 172.21.15.5:0/1547583324 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f171c004da0 con 0x7f17340f46d0 2021-03-09T08:58:55.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.011+0000 7f172affd700 1 -- 172.21.15.5:0/1547583324 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f1730013740 con 0x7f17340f46d0 2021-03-09T08:58:55.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.011+0000 7f172affd700 1 --2- 172.21.15.5:0/1547583324 >> v2:172.21.15.5:6800/4269212764 conn(0x7f172005c210 0x7f172005e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:55.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.011+0000 7f172affd700 1 -- 172.21.15.5:0/1547583324 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1484+0+0 (secure 0 0 0) 0x7f1730076a80 con 0x7f17340f46d0 2021-03-09T08:58:55.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.011+0000 7f1739be5700 1 --2- 172.21.15.5:0/1547583324 >> v2:172.21.15.5:6800/4269212764 conn(0x7f172005c210 0x7f172005e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:55.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.011+0000 7f1739be5700 1 --2- 172.21.15.5:0/1547583324 >> v2:172.21.15.5:6800/4269212764 conn(0x7f172005c210 0x7f172005e6d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1724009fd0 tx=0x7f172400b040).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:55.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.015+0000 7f172affd700 1 -- 172.21.15.5:0/1547583324 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f173004b4e0 con 0x7f17340f46d0 2021-03-09T08:58:55.062 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:54 smithi005 bash[16589]: cluster 2021-03-09T08:58:53.301971+0000 mgr.y (mgr.14144) 78 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:55.062 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:54 smithi005 bash[16589]: audit 2021-03-09T08:58:53.962814+0000 mon.a (mon.0) 207 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T08:58:55.063 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:54 smithi005 bash[16589]: cluster 2021-03-09T08:58:53.963075+0000 mon.a (mon.0) 208 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-03-09T08:58:55.063 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:54 smithi005 bash[16589]: audit 2021-03-09T08:58:53.963884+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-09T08:58:55.063 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:54 smithi005 bash[16589]: audit 2021-03-09T08:58:53.972331+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-09T08:58:55.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:54 smithi005 bash[12738]: cluster 2021-03-09T08:58:53.301971+0000 mgr.y (mgr.14144) 78 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:55.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:54 smithi005 bash[12738]: audit 2021-03-09T08:58:53.962814+0000 mon.a (mon.0) 207 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T08:58:55.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:54 smithi005 bash[12738]: cluster 2021-03-09T08:58:53.963075+0000 mon.a (mon.0) 208 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-03-09T08:58:55.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:54 smithi005 bash[12738]: audit 2021-03-09T08:58:53.963884+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-09T08:58:55.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:54 smithi005 bash[12738]: audit 2021-03-09T08:58:53.972331+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-09T08:58:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:58:55.239+0000 7f173b3e8700 1 -- 172.21.15.5:0/1547583324 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f171c000bc0 con 0x7f172005c210 2021-03-09T08:58:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:54 smithi146 bash[13136]: cluster 2021-03-09T08:58:53.301971+0000 mgr.y (mgr.14144) 78 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:55.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:54 smithi146 bash[13136]: audit 2021-03-09T08:58:53.962814+0000 mon.a (mon.0) 207 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T08:58:55.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:54 smithi146 bash[13136]: cluster 2021-03-09T08:58:53.963075+0000 mon.a (mon.0) 208 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-03-09T08:58:55.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:54 smithi146 bash[13136]: audit 2021-03-09T08:58:53.963884+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-09T08:58:55.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:54 smithi146 bash[13136]: audit 2021-03-09T08:58:53.972331+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-09T08:58:55.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:55 smithi005 bash[12738]: cluster 2021-03-09T08:58:53.123882+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-09T08:58:55.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:55 smithi005 bash[12738]: cluster 2021-03-09T08:58:53.124129+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-09T08:58:55.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:55 smithi005 bash[12738]: cluster 2021-03-09T08:58:54.970888+0000 mon.a (mon.0) 211 : cluster [INF] osd.0 v2:172.21.15.5:6801/2804368853 boot 2021-03-09T08:58:55.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:55 smithi005 bash[12738]: cluster 2021-03-09T08:58:54.970941+0000 mon.a (mon.0) 212 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2021-03-09T08:58:55.987 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:55 smithi005 bash[12738]: audit 2021-03-09T08:58:54.972179+0000 mon.a (mon.0) 213 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-09T08:58:55.987 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:55 smithi005 bash[12738]: audit 2021-03-09T08:58:55.244708+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-09T08:58:55.987 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:55 smithi005 bash[12738]: audit 2021-03-09T08:58:55.248050+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-09T08:58:55.987 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:55 smithi005 bash[12738]: audit 2021-03-09T08:58:55.249564+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:56.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:55 smithi146 bash[13136]: cluster 2021-03-09T08:58:53.123882+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-09T08:58:56.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:55 smithi146 bash[13136]: cluster 2021-03-09T08:58:53.124129+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-09T08:58:56.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:55 smithi146 bash[13136]: cluster 2021-03-09T08:58:54.970888+0000 mon.a (mon.0) 211 : cluster [INF] osd.0 v2:172.21.15.5:6801/2804368853 boot 2021-03-09T08:58:56.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:55 smithi146 bash[13136]: cluster 2021-03-09T08:58:54.970941+0000 mon.a (mon.0) 212 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2021-03-09T08:58:56.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:55 smithi146 bash[13136]: audit 2021-03-09T08:58:54.972179+0000 mon.a (mon.0) 213 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-09T08:58:56.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:55 smithi146 bash[13136]: audit 2021-03-09T08:58:55.244708+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-09T08:58:56.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:55 smithi146 bash[13136]: audit 2021-03-09T08:58:55.248050+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-09T08:58:56.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:55 smithi146 bash[13136]: audit 2021-03-09T08:58:55.249564+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:56.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:55 smithi005 bash[16589]: cluster 2021-03-09T08:58:53.123882+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-09T08:58:56.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:55 smithi005 bash[16589]: cluster 2021-03-09T08:58:53.124129+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-09T08:58:56.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:55 smithi005 bash[16589]: cluster 2021-03-09T08:58:54.970888+0000 mon.a (mon.0) 211 : cluster [INF] osd.0 v2:172.21.15.5:6801/2804368853 boot 2021-03-09T08:58:56.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:55 smithi005 bash[16589]: cluster 2021-03-09T08:58:54.970941+0000 mon.a (mon.0) 212 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2021-03-09T08:58:56.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:55 smithi005 bash[16589]: audit 2021-03-09T08:58:54.972179+0000 mon.a (mon.0) 213 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-09T08:58:56.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:55 smithi005 bash[16589]: audit 2021-03-09T08:58:55.244708+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-09T08:58:56.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:55 smithi005 bash[16589]: audit 2021-03-09T08:58:55.248050+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-09T08:58:56.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:55 smithi005 bash[16589]: audit 2021-03-09T08:58:55.249564+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:58:57.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:56 smithi146 bash[13136]: audit 2021-03-09T08:58:55.243143+0000 mgr.y (mgr.14144) 79 : audit [DBG] from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:58:57.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:56 smithi146 bash[13136]: cephadm 2021-03-09T08:58:55.247509+0000 mgr.y (mgr.14144) 80 : cephadm [INF] Applying service osd.None on host smithi005... 2021-03-09T08:58:57.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:56 smithi146 bash[13136]: cluster 2021-03-09T08:58:55.302392+0000 mgr.y (mgr.14144) 81 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:57.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:56 smithi146 bash[13136]: cluster 2021-03-09T08:58:55.988556+0000 mon.a (mon.0) 217 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-03-09T08:58:57.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:56 smithi146 bash[13136]: audit 2021-03-09T08:58:56.300938+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:57.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:56 smithi146 bash[13136]: audit 2021-03-09T08:58:56.307451+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:57.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:56 smithi146 bash[13136]: audit 2021-03-09T08:58:56.309241+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:57.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:56 smithi146 bash[13136]: audit 2021-03-09T08:58:56.315581+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:57.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:56 smithi005 bash[12738]: audit 2021-03-09T08:58:55.243143+0000 mgr.y (mgr.14144) 79 : audit [DBG] from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:58:57.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:56 smithi005 bash[12738]: cephadm 2021-03-09T08:58:55.247509+0000 mgr.y (mgr.14144) 80 : cephadm [INF] Applying service osd.None on host smithi005... 2021-03-09T08:58:57.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:56 smithi005 bash[12738]: cluster 2021-03-09T08:58:55.302392+0000 mgr.y (mgr.14144) 81 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:57.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:56 smithi005 bash[12738]: cluster 2021-03-09T08:58:55.988556+0000 mon.a (mon.0) 217 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-03-09T08:58:57.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:56 smithi005 bash[12738]: audit 2021-03-09T08:58:56.300938+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:57.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:56 smithi005 bash[12738]: audit 2021-03-09T08:58:56.307451+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:57.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:56 smithi005 bash[12738]: audit 2021-03-09T08:58:56.309241+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:57.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:56 smithi005 bash[12738]: audit 2021-03-09T08:58:56.315581+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:57.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:56 smithi005 bash[16589]: audit 2021-03-09T08:58:55.243143+0000 mgr.y (mgr.14144) 79 : audit [DBG] from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:58:57.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:56 smithi005 bash[16589]: cephadm 2021-03-09T08:58:55.247509+0000 mgr.y (mgr.14144) 80 : cephadm [INF] Applying service osd.None on host smithi005... 2021-03-09T08:58:57.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:56 smithi005 bash[16589]: cluster 2021-03-09T08:58:55.302392+0000 mgr.y (mgr.14144) 81 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-09T08:58:57.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:56 smithi005 bash[16589]: cluster 2021-03-09T08:58:55.988556+0000 mon.a (mon.0) 217 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-03-09T08:58:57.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:56 smithi005 bash[16589]: audit 2021-03-09T08:58:56.300938+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:57.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:56 smithi005 bash[16589]: audit 2021-03-09T08:58:56.307451+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:57.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:56 smithi005 bash[16589]: audit 2021-03-09T08:58:56.309241+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:58:57.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:56 smithi005 bash[16589]: audit 2021-03-09T08:58:56.315581+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:58:58.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:57 smithi146 bash[13136]: cephadm 2021-03-09T08:58:56.292987+0000 mgr.y (mgr.14144) 82 : cephadm [INF] Detected new or changed devices on smithi005 2021-03-09T08:58:58.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:57 smithi005 bash[12738]: cephadm 2021-03-09T08:58:56.292987+0000 mgr.y (mgr.14144) 82 : cephadm [INF] Detected new or changed devices on smithi005 2021-03-09T08:58:58.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:57 smithi005 bash[16589]: cephadm 2021-03-09T08:58:56.292987+0000 mgr.y (mgr.14144) 82 : cephadm [INF] Detected new or changed devices on smithi005 2021-03-09T08:58:59.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:58:58 smithi146 bash[13136]: cluster 2021-03-09T08:58:57.302892+0000 mgr.y (mgr.14144) 83 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:58:59.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:58:58 smithi005 bash[12738]: cluster 2021-03-09T08:58:57.302892+0000 mgr.y (mgr.14144) 83 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:58:59.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:58:58 smithi005 bash[16589]: cluster 2021-03-09T08:58:57.302892+0000 mgr.y (mgr.14144) 83 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:00 smithi146 bash[13136]: audit 2021-03-09T08:58:59.685036+0000 mon.c (mon.1) 8 : audit [INF] from='client.? 172.21.15.5:0/183320810' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6863a27-13de-4b82-b69b-efe93278a155"}]: dispatch 2021-03-09T08:59:00.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:00 smithi146 bash[13136]: audit 2021-03-09T08:58:59.685737+0000 mon.a (mon.0) 222 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6863a27-13de-4b82-b69b-efe93278a155"}]: dispatch 2021-03-09T08:59:00.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:00 smithi146 bash[13136]: audit 2021-03-09T08:58:59.693381+0000 mon.a (mon.0) 223 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6863a27-13de-4b82-b69b-efe93278a155"}]': finished 2021-03-09T08:59:00.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:00 smithi146 bash[13136]: cluster 2021-03-09T08:58:59.693607+0000 mon.a (mon.0) 224 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2021-03-09T08:59:00.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:00 smithi146 bash[13136]: audit 2021-03-09T08:58:59.693874+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-09T08:59:00.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:00 smithi005 bash[12738]: audit 2021-03-09T08:58:59.685036+0000 mon.c (mon.1) 8 : audit [INF] from='client.? 172.21.15.5:0/183320810' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6863a27-13de-4b82-b69b-efe93278a155"}]: dispatch 2021-03-09T08:59:00.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:00 smithi005 bash[12738]: audit 2021-03-09T08:58:59.685737+0000 mon.a (mon.0) 222 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6863a27-13de-4b82-b69b-efe93278a155"}]: dispatch 2021-03-09T08:59:00.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:00 smithi005 bash[12738]: audit 2021-03-09T08:58:59.693381+0000 mon.a (mon.0) 223 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6863a27-13de-4b82-b69b-efe93278a155"}]': finished 2021-03-09T08:59:00.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:00 smithi005 bash[12738]: cluster 2021-03-09T08:58:59.693607+0000 mon.a (mon.0) 224 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2021-03-09T08:59:00.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:00 smithi005 bash[12738]: audit 2021-03-09T08:58:59.693874+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-09T08:59:00.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:00 smithi005 bash[16589]: audit 2021-03-09T08:58:59.685036+0000 mon.c (mon.1) 8 : audit [INF] from='client.? 172.21.15.5:0/183320810' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6863a27-13de-4b82-b69b-efe93278a155"}]: dispatch 2021-03-09T08:59:00.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:00 smithi005 bash[16589]: audit 2021-03-09T08:58:59.685737+0000 mon.a (mon.0) 222 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6863a27-13de-4b82-b69b-efe93278a155"}]: dispatch 2021-03-09T08:59:00.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:00 smithi005 bash[16589]: audit 2021-03-09T08:58:59.693381+0000 mon.a (mon.0) 223 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6863a27-13de-4b82-b69b-efe93278a155"}]': finished 2021-03-09T08:59:00.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:00 smithi005 bash[16589]: cluster 2021-03-09T08:58:59.693607+0000 mon.a (mon.0) 224 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2021-03-09T08:59:00.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:00 smithi005 bash[16589]: audit 2021-03-09T08:58:59.693874+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-09T08:59:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:01 smithi146 bash[13136]: cluster 2021-03-09T08:58:59.303228+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:01.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:01 smithi146 bash[13136]: audit 2021-03-09T08:59:00.419404+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.5:0/292326115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-09T08:59:01.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:01 smithi005 bash[12738]: cluster 2021-03-09T08:58:59.303228+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:01.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:01 smithi005 bash[12738]: audit 2021-03-09T08:59:00.419404+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.5:0/292326115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-09T08:59:01.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:01 smithi005 bash[16589]: cluster 2021-03-09T08:58:59.303228+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:01.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:01 smithi005 bash[16589]: audit 2021-03-09T08:59:00.419404+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.5:0/292326115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-09T08:59:03.243 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:02 smithi005 bash[16589]: cluster 2021-03-09T08:59:01.303669+0000 mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:03.243 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:02 smithi005 bash[12738]: cluster 2021-03-09T08:59:01.303669+0000 mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:03.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:02 smithi146 bash[13136]: cluster 2021-03-09T08:59:01.303669+0000 mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:05.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:04 smithi005 bash[12738]: cluster 2021-03-09T08:59:03.304208+0000 mgr.y (mgr.14144) 86 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:05.164 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:04 smithi005 bash[16589]: cluster 2021-03-09T08:59:03.304208+0000 mgr.y (mgr.14144) 86 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:05.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:04 smithi146 bash[13136]: cluster 2021-03-09T08:59:03.304208+0000 mgr.y (mgr.14144) 86 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:07.116 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:06 smithi005 bash[16589]: cluster 2021-03-09T08:59:05.304830+0000 mgr.y (mgr.14144) 87 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:07.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:06 smithi005 bash[12738]: cluster 2021-03-09T08:59:05.304830+0000 mgr.y (mgr.14144) 87 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:07.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:06 smithi146 bash[13136]: cluster 2021-03-09T08:59:05.304830+0000 mgr.y (mgr.14144) 87 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:09.225 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:08 smithi005 bash[16589]: cluster 2021-03-09T08:59:07.305373+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:09.225 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:08 smithi005 bash[16589]: audit 2021-03-09T08:59:07.992680+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-03-09T08:59:09.225 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:08 smithi005 bash[16589]: audit 2021-03-09T08:59:07.994126+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:59:09.226 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:08 smithi005 bash[16589]: cephadm 2021-03-09T08:59:07.995380+0000 mgr.y (mgr.14144) 89 : cephadm [INF] Deploying daemon osd.1 on smithi005 2021-03-09T08:59:09.226 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:08 smithi005 bash[12738]: cluster 2021-03-09T08:59:07.305373+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:09.226 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:08 smithi005 bash[12738]: audit 2021-03-09T08:59:07.992680+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-03-09T08:59:09.226 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:08 smithi005 bash[12738]: audit 2021-03-09T08:59:07.994126+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:59:09.227 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:08 smithi005 bash[12738]: cephadm 2021-03-09T08:59:07.995380+0000 mgr.y (mgr.14144) 89 : cephadm [INF] Deploying daemon osd.1 on smithi005 2021-03-09T08:59:09.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:08 smithi146 bash[13136]: cluster 2021-03-09T08:59:07.305373+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:09.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:08 smithi146 bash[13136]: audit 2021-03-09T08:59:07.992680+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-03-09T08:59:09.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:08 smithi146 bash[13136]: audit 2021-03-09T08:59:07.994126+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:59:09.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:08 smithi146 bash[13136]: cephadm 2021-03-09T08:59:07.995380+0000 mgr.y (mgr.14144) 89 : cephadm [INF] Deploying daemon osd.1 on smithi005 2021-03-09T08:59:10.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:10.555+0000 7f172affd700 1 -- 172.21.15.5:0/1547583324 <== mgr.14144 v2:172.21.15.5:6800/4269212764 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f171c000bc0 con 0x7f172005c210 2021-03-09T08:59:10.555 INFO:teuthology.orchestra.run.smithi005.stdout:Created osd(s) 1 on host 'smithi005' 2021-03-09T08:59:10.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:10.555+0000 7f173b3e8700 1 -- 172.21.15.5:0/1547583324 >> v2:172.21.15.5:6800/4269212764 conn(0x7f172005c210 msgr2=0x7f172005e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:59:10.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:10.555+0000 7f173b3e8700 1 --2- 172.21.15.5:0/1547583324 >> v2:172.21.15.5:6800/4269212764 conn(0x7f172005c210 0x7f172005e6d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1724009fd0 tx=0x7f172400b040).stop 2021-03-09T08:59:10.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:10.555+0000 7f173b3e8700 1 -- 172.21.15.5:0/1547583324 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17340f46d0 msgr2=0x7f17340f6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:59:10.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:10.555+0000 7f173b3e8700 1 --2- 172.21.15.5:0/1547583324 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17340f46d0 0x7f17340f6f00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f173000b690 tx=0x7f1730009b30).stop 2021-03-09T08:59:10.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:10.555+0000 7f173b3e8700 1 -- 172.21.15.5:0/1547583324 shutdown_connections 2021-03-09T08:59:10.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:10.555+0000 7f173b3e8700 1 --2- 172.21.15.5:0/1547583324 >> v2:172.21.15.5:6800/4269212764 conn(0x7f172005c210 0x7f172005e6d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:10.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:10.555+0000 7f173b3e8700 1 --2- 172.21.15.5:0/1547583324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17340f7fd0 0x7f17340f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:10.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:10.555+0000 7f173b3e8700 1 --2- 172.21.15.5:0/1547583324 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f17340f46d0 0x7f17340f6f00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:10.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:10.555+0000 7f173b3e8700 1 --2- 172.21.15.5:0/1547583324 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17340f7630 0x7f17340f5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:10.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:10.555+0000 7f173b3e8700 1 -- 172.21.15.5:0/1547583324 >> 172.21.15.5:0/1547583324 conn(0x7f17340efe90 msgr2=0x7f17340f1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:59:10.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:10.555+0000 7f173b3e8700 1 -- 172.21.15.5:0/1547583324 shutdown_connections 2021-03-09T08:59:10.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:10.555+0000 7f173b3e8700 1 -- 172.21.15.5:0/1547583324 wait complete. 2021-03-09T08:59:11.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:10 smithi005 bash[12738]: cluster 2021-03-09T08:59:09.306260+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:11.226 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:10 smithi005 bash[12738]: audit 2021-03-09T08:59:10.541810+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:59:11.226 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:10 smithi005 bash[12738]: audit 2021-03-09T08:59:10.547890+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:11.226 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:10 smithi005 bash[12738]: audit 2021-03-09T08:59:10.555635+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:11.227 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:10 smithi005 bash[16589]: cluster 2021-03-09T08:59:09.306260+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:11.227 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:10 smithi005 bash[16589]: audit 2021-03-09T08:59:10.541810+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:59:11.227 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:10 smithi005 bash[16589]: audit 2021-03-09T08:59:10.547890+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:11.227 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:10 smithi005 bash[16589]: audit 2021-03-09T08:59:10.555635+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:11.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:10 smithi146 bash[13136]: cluster 2021-03-09T08:59:09.306260+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:11.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:10 smithi146 bash[13136]: audit 2021-03-09T08:59:10.541810+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:59:11.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:10 smithi146 bash[13136]: audit 2021-03-09T08:59:10.547890+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:11.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:10 smithi146 bash[13136]: audit 2021-03-09T08:59:10.555635+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:12.006 DEBUG:teuthology.orchestra.run.smithi005:osd.1> sudo journalctl -f -n 0 -u ceph-252abbe0-80b5-11eb-907b-001a4aab830c@osd.1.service 2021-03-09T08:59:12.012 INFO:tasks.cephadm:Deploying osd.2 on smithi005 with /dev/vg_nvme/lv_2... 2021-03-09T08:59:12.012 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-03-09T08:59:12.077 INFO:journalctl@ceph.osd.1.smithi005.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-09T08:59:13.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:12 smithi146 bash[13136]: cluster 2021-03-09T08:59:11.306786+0000 mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:13.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:12 smithi005 bash[12738]: cluster 2021-03-09T08:59:11.306786+0000 mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:13.330 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:12 smithi005 bash[16589]: cluster 2021-03-09T08:59:11.306786+0000 mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:13.330 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:13 smithi005 bash[21453]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-03-09T08:59:13.331 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:13 smithi005 bash[21453]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2021-03-09T08:59:13.331 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:13 smithi005 bash[21453]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2021-03-09T08:59:13.331 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:13 smithi005 bash[21453]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2021-03-09T08:59:13.331 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:13 smithi005 bash[21453]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2021-03-09T08:59:13.331 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:13 smithi005 bash[21453]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-03-09T08:59:13.332 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:13 smithi005 bash[21453]: --> ceph-volume lvm activate successful for osd ID: 1 2021-03-09T08:59:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:14 smithi146 bash[13136]: cluster 2021-03-09T08:59:13.307282+0000 mgr.y (mgr.14144) 92 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:15.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:14 smithi005 bash[12738]: cluster 2021-03-09T08:59:13.307282+0000 mgr.y (mgr.14144) 92 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:15.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:14 smithi005 bash[16589]: cluster 2021-03-09T08:59:13.307282+0000 mgr.y (mgr.14144) 92 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:16.108 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54d93f1700 10 -- 172.21.15.5:0/740940743 dispatch_throttle_release 413 to dispatch throttler 413/104857600 2021-03-09T08:59:16.109 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54d93f1700 20 -- 172.21.15.5:0/740940743 done calling dispatch on 0x560b5f73d4a0 2021-03-09T08:59:16.109 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 20 -- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 msgr2=0x560b5f72e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2021-03-09T08:59:16.109 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 20 -- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 msgr2=0x560b5f72e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=96 2021-03-09T08:59:16.109 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 20 --2- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 0x560b5f72e300 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x560b5f741f50 tx=0x560b67974d80).handle_read_frame_preamble_main r=0 2021-03-09T08:59:16.110 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 20 --2- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 0x560b5f72e300 secure :-1 s=THROTTLE_MESSAGE pgs=14 cs=0 l=1 rev1=1 rx=0x560b5f741f50 tx=0x560b67974d80).throttle_message 2021-03-09T08:59:16.110 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 20 --2- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 0x560b5f72e300 secure :-1 s=THROTTLE_BYTES pgs=14 cs=0 l=1 rev1=1 rx=0x560b5f741f50 tx=0x560b67974d80).throttle_bytes 2021-03-09T08:59:16.110 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 20 --2- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 0x560b5f72e300 secure :-1 s=THROTTLE_DISPATCH_QUEUE pgs=14 cs=0 l=1 rev1=1 rx=0x560b5f741f50 tx=0x560b67974d80).throttle_dispatch_queue 2021-03-09T08:59:16.110 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 20 --2- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 0x560b5f72e300 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rev1=1 rx=0x560b5f741f50 tx=0x560b67974d80).read_frame_segment seg_idx=0 2021-03-09T08:59:16.110 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 20 --2- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 0x560b5f72e300 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rev1=1 rx=0x560b5f741f50 tx=0x560b67974d80).read_frame_segment seg_idx=1 2021-03-09T08:59:16.111 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 20 -- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 msgr2=0x560b5f72e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=208 2021-03-09T08:59:16.111 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 20 --2- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 0x560b5f72e300 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rev1=1 rx=0x560b5f741f50 tx=0x560b67974d80).handle_read_frame_segment r=0 2021-03-09T08:59:16.111 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 20 -- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 msgr2=0x560b5f72e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2021-03-09T08:59:16.111 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 20 --2- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 0x560b5f72e300 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rev1=1 rx=0x560b5f741f50 tx=0x560b67974d80).handle_read_frame_epilogue_main r=0 2021-03-09T08:59:16.111 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 10 --2- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 0x560b5f72e300 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rev1=1 rx=0x560b5f741f50 tx=0x560b67974d80).handle_read_frame_dispatch tag=17 2021-03-09T08:59:16.112 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 20 --2- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 0x560b5f72e300 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rev1=1 rx=0x560b5f741f50 tx=0x560b67974d80).handle_message 2021-03-09T08:59:16.112 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 5 --2- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 0x560b5f72e300 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rev1=1 rx=0x560b5f741f50 tx=0x560b67974d80).handle_message got 194 + 0 + 0 byte message. envelope type=18 src mon.1 off 0 2021-03-09T08:59:16.112 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 5 --2- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 0x560b5f72e300 secure :-1 s=READ_MESSAGE_COMPLETE pgs=14 cs=0 l=1 rev1=1 rx=0x560b5f741f50 tx=0x560b67974d80).handle_message received message m=0x560b679444e0 seq=4 from=mon.1 type=18 auth_reply(proto 2 0 (0) Success) v1 2021-03-09T08:59:16.112 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 20 -- 172.21.15.5:0/740940743 queue 0x560b679444e0 prio 196 2021-03-09T08:59:16.113 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 20 --2- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 0x560b5f72e300 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x560b5f741f50 tx=0x560b67974d80).read_frame 2021-03-09T08:59:16.113 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54dabf4700 20 -- 172.21.15.5:0/740940743 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x560b679c1400 msgr2=0x560b5f72e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=96 2021-03-09T08:59:16.113 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54d93f1700 1 -- 172.21.15.5:0/740940743 <== mon.1 v2:172.21.15.5:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560b679444e0 con 0x560b679c1400 2021-03-09T08:59:16.113 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54d93f1700 10 -- 172.21.15.5:0/740940743 dispatch_throttle_release 194 to dispatch throttler 194/104857600 2021-03-09T08:59:16.113 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:15 smithi005 bash[21659]: debug 2021-03-09T08:59:15.855+0000 7f54d93f1700 20 -- 172.21.15.5:0/740940743 done calling dispatch on 0x560b679444e0 2021-03-09T08:59:16.482 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:16 smithi005 bash[21659]: debug 2021-03-09T08:59:16.155+0000 7f54e0e86080 -1 Falling back to public interface 2021-03-09T08:59:16.792 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-03-09T08:59:16.793 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-03-09T08:59:16.793 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10+0 records in 2021-03-09T08:59:16.793 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2021-03-09T08:59:16.794 INFO:teuthology.orchestra.run.smithi005.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0142953 s, 734 MB/s 2021-03-09T08:59:16.794 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping successful for: 2021-03-09T08:59:17.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:16 smithi005 bash[12738]: cluster 2021-03-09T08:59:15.307962+0000 mgr.y (mgr.14144) 93 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:17.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:16 smithi005 bash[12738]: audit 2021-03-09T08:59:16.109525+0000 mon.a (mon.0) 231 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:17.117 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:16 smithi005 bash[16589]: cluster 2021-03-09T08:59:15.307962+0000 mgr.y (mgr.14144) 93 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:17.117 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:16 smithi005 bash[16589]: audit 2021-03-09T08:59:16.109525+0000 mon.a (mon.0) 231 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:17.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:16 smithi146 bash[13136]: cluster 2021-03-09T08:59:15.307962+0000 mgr.y (mgr.14144) 93 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:17.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:16 smithi146 bash[13136]: audit 2021-03-09T08:59:16.109525+0000 mon.a (mon.0) 231 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:17.855 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph orch daemon add osd smithi005:vg_nvme/lv_2 2021-03-09T08:59:19.185 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:18 smithi005 bash[12738]: cluster 2021-03-09T08:59:17.308509+0000 mgr.y (mgr.14144) 94 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:19.186 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:18 smithi005 bash[16589]: cluster 2021-03-09T08:59:17.308509+0000 mgr.y (mgr.14144) 94 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:19.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:18 smithi146 bash[13136]: cluster 2021-03-09T08:59:17.308509+0000 mgr.y (mgr.14144) 94 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:19.522 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:19 smithi005 bash[21659]: debug 2021-03-09T08:59:19.183+0000 7f54e0e86080 -1 osd.1 0 log_to_monitors {default=true} 2021-03-09T08:59:20.112 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:19 smithi005 bash[12738]: audit 2021-03-09T08:59:19.193357+0000 mon.a (mon.0) 232 : audit [INF] from='osd.1 v2:172.21.15.5:6805/137257932' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-03-09T08:59:20.112 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:19 smithi005 bash[16589]: audit 2021-03-09T08:59:19.193357+0000 mon.a (mon.0) 232 : audit [INF] from='osd.1 v2:172.21.15.5:6805/137257932' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-03-09T08:59:20.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:19 smithi146 bash[13136]: audit 2021-03-09T08:59:19.193357+0000 mon.a (mon.0) 232 : audit [INF] from='osd.1 v2:172.21.15.5:6805/137257932' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-03-09T08:59:21.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:20 smithi146 bash[13136]: cluster 2021-03-09T08:59:19.309009+0000 mgr.y (mgr.14144) 95 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:21.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:20 smithi146 bash[13136]: audit 2021-03-09T08:59:19.943267+0000 mon.a (mon.0) 233 : audit [INF] from='osd.1 v2:172.21.15.5:6805/137257932' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-03-09T08:59:21.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:20 smithi146 bash[13136]: cluster 2021-03-09T08:59:19.943372+0000 mon.a (mon.0) 234 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-03-09T08:59:21.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:20 smithi146 bash[13136]: audit 2021-03-09T08:59:19.943531+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-09T08:59:21.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:20 smithi146 bash[13136]: audit 2021-03-09T08:59:19.944288+0000 mon.a (mon.0) 236 : audit [INF] from='osd.1 v2:172.21.15.5:6805/137257932' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2021-03-09T08:59:21.366 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 09 08:59:20 smithi005 bash[21659]: debug 2021-03-09T08:59:20.951+0000 7f54d73ed700 -1 osd.1 0 waiting for initial osdmap 2021-03-09T08:59:21.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:20 smithi005 bash[12738]: cluster 2021-03-09T08:59:19.309009+0000 mgr.y (mgr.14144) 95 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:21.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:20 smithi005 bash[12738]: audit 2021-03-09T08:59:19.943267+0000 mon.a (mon.0) 233 : audit [INF] from='osd.1 v2:172.21.15.5:6805/137257932' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-03-09T08:59:21.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:20 smithi005 bash[12738]: cluster 2021-03-09T08:59:19.943372+0000 mon.a (mon.0) 234 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-03-09T08:59:21.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:20 smithi005 bash[12738]: audit 2021-03-09T08:59:19.943531+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-09T08:59:21.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:20 smithi005 bash[12738]: audit 2021-03-09T08:59:19.944288+0000 mon.a (mon.0) 236 : audit [INF] from='osd.1 v2:172.21.15.5:6805/137257932' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2021-03-09T08:59:21.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:20 smithi005 bash[16589]: cluster 2021-03-09T08:59:19.309009+0000 mgr.y (mgr.14144) 95 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:21.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:20 smithi005 bash[16589]: audit 2021-03-09T08:59:19.943267+0000 mon.a (mon.0) 233 : audit [INF] from='osd.1 v2:172.21.15.5:6805/137257932' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-03-09T08:59:21.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:20 smithi005 bash[16589]: cluster 2021-03-09T08:59:19.943372+0000 mon.a (mon.0) 234 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-03-09T08:59:21.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:20 smithi005 bash[16589]: audit 2021-03-09T08:59:19.943531+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-09T08:59:21.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:20 smithi005 bash[16589]: audit 2021-03-09T08:59:19.944288+0000 mon.a (mon.0) 236 : audit [INF] from='osd.1 v2:172.21.15.5:6805/137257932' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2021-03-09T08:59:22.261 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:21 smithi005 bash[12738]: audit 2021-03-09T08:59:20.948067+0000 mon.a (mon.0) 237 : audit [INF] from='osd.1 v2:172.21.15.5:6805/137257932' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T08:59:22.261 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:21 smithi005 bash[12738]: cluster 2021-03-09T08:59:20.948327+0000 mon.a (mon.0) 238 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-03-09T08:59:22.261 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:21 smithi005 bash[12738]: audit 2021-03-09T08:59:20.949143+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-09T08:59:22.262 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:21 smithi005 bash[12738]: audit 2021-03-09T08:59:20.956312+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-09T08:59:22.262 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:21 smithi005 bash[16589]: audit 2021-03-09T08:59:20.948067+0000 mon.a (mon.0) 237 : audit [INF] from='osd.1 v2:172.21.15.5:6805/137257932' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T08:59:22.262 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:21 smithi005 bash[16589]: cluster 2021-03-09T08:59:20.948327+0000 mon.a (mon.0) 238 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-03-09T08:59:22.263 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:21 smithi005 bash[16589]: audit 2021-03-09T08:59:20.949143+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-09T08:59:22.263 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:21 smithi005 bash[16589]: audit 2021-03-09T08:59:20.956312+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-09T08:59:22.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:21 smithi146 bash[13136]: audit 2021-03-09T08:59:20.948067+0000 mon.a (mon.0) 237 : audit [INF] from='osd.1 v2:172.21.15.5:6805/137257932' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T08:59:22.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:21 smithi146 bash[13136]: cluster 2021-03-09T08:59:20.948327+0000 mon.a (mon.0) 238 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-03-09T08:59:22.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:21 smithi146 bash[13136]: audit 2021-03-09T08:59:20.949143+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-09T08:59:22.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:21 smithi146 bash[13136]: audit 2021-03-09T08:59:20.956312+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-09T08:59:22.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 -- 172.21.15.5:0/3271212572 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd82c0f6b10 msgr2=0x7fd82c0fd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:59:22.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 --2- 172.21.15.5:0/3271212572 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd82c0f6b10 0x7fd82c0fd410 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fd824009df0 tx=0x7fd824004d10).stop 2021-03-09T08:59:22.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 -- 172.21.15.5:0/3271212572 shutdown_connections 2021-03-09T08:59:22.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 --2- 172.21.15.5:0/3271212572 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd82c0f6b10 0x7fd82c0fd410 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:22.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 --2- 172.21.15.5:0/3271212572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd82c0f4f20 0x7fd82c0f5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:22.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 --2- 172.21.15.5:0/3271212572 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82c0f61c0 0x7fd82c0f6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:22.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 -- 172.21.15.5:0/3271212572 >> 172.21.15.5:0/3271212572 conn(0x7fd82c0f0700 msgr2=0x7fd82c0f2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:59:22.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 -- 172.21.15.5:0/3271212572 shutdown_connections 2021-03-09T08:59:22.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 -- 172.21.15.5:0/3271212572 wait complete. 2021-03-09T08:59:22.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 Processor -- start 2021-03-09T08:59:22.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 -- start start 2021-03-09T08:59:22.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd82c0f4f20 0x7fd82c104d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:22.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd82c0f61c0 0x7fd82c105280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:22.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82c0f6b10 0x7fd82c106090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:22.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c106660 con 0x7fd82c0f6b10 2021-03-09T08:59:22.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c1067d0 con 0x7fd82c0f4f20 2021-03-09T08:59:22.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c106990 con 0x7fd82c0f61c0 2021-03-09T08:59:22.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82ad9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82c0f6b10 0x7fd82c106090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:22.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82ad9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82c0f6b10 0x7fd82c106090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:46702/0 (socket says 172.21.15.5:46702) 2021-03-09T08:59:22.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82ad9d700 1 -- 172.21.15.5:0/2507589526 learned_addr learned my addr 172.21.15.5:0/2507589526 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:59:22.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd82a59c700 1 --2- 172.21.15.5:0/2507589526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd82c0f4f20 0x7fd82c104d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:22.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.531+0000 7fd829d9b700 1 --2- 172.21.15.5:0/2507589526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd82c0f61c0 0x7fd82c105280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:22.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.535+0000 7fd82ad9d700 1 -- 172.21.15.5:0/2507589526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd82c0f61c0 msgr2=0x7fd82c105280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:59:22.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.535+0000 7fd82ad9d700 1 --2- 172.21.15.5:0/2507589526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd82c0f61c0 0x7fd82c105280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:22.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.535+0000 7fd82ad9d700 1 -- 172.21.15.5:0/2507589526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd82c0f4f20 msgr2=0x7fd82c104d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:59:22.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.535+0000 7fd82ad9d700 1 --2- 172.21.15.5:0/2507589526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd82c0f4f20 0x7fd82c104d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:22.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.535+0000 7fd82ad9d700 1 -- 172.21.15.5:0/2507589526 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82c106b50 con 0x7fd82c0f6b10 2021-03-09T08:59:22.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.535+0000 7fd82ad9d700 1 --2- 172.21.15.5:0/2507589526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82c0f6b10 0x7fd82c106090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fd824013750 tx=0x7fd824004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:22.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.535+0000 7fd81affd700 1 -- 172.21.15.5:0/2507589526 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82400e030 con 0x7fd82c0f6b10 2021-03-09T08:59:22.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.535+0000 7fd82b59e700 1 -- 172.21.15.5:0/2507589526 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd82c106da0 con 0x7fd82c0f6b10 2021-03-09T08:59:22.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.535+0000 7fd82b59e700 1 -- 172.21.15.5:0/2507589526 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82c05f3a0 con 0x7fd82c0f6b10 2021-03-09T08:59:22.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.535+0000 7fd81affd700 1 -- 172.21.15.5:0/2507589526 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7fd824004e90 con 0x7fd82c0f6b10 2021-03-09T08:59:22.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.535+0000 7fd81affd700 1 -- 172.21.15.5:0/2507589526 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824018070 con 0x7fd82c0f6b10 2021-03-09T08:59:22.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.535+0000 7fd82b59e700 1 -- 172.21.15.5:0/2507589526 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd82c0f4310 con 0x7fd82c0f6b10 2021-03-09T08:59:22.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.535+0000 7fd81affd700 1 -- 172.21.15.5:0/2507589526 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7fd824013930 con 0x7fd82c0f6b10 2021-03-09T08:59:22.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.535+0000 7fd81affd700 1 --2- 172.21.15.5:0/2507589526 >> v2:172.21.15.5:6800/4269212764 conn(0x7fd81405c210 0x7fd81405e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:22.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.539+0000 7fd81affd700 1 -- 172.21.15.5:0/2507589526 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 1776+0+0 (secure 0 0 0) 0x7fd824077110 con 0x7fd82c0f6b10 2021-03-09T08:59:22.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.539+0000 7fd82a59c700 1 --2- 172.21.15.5:0/2507589526 >> v2:172.21.15.5:6800/4269212764 conn(0x7fd81405c210 0x7fd81405e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:22.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.539+0000 7fd82a59c700 1 --2- 172.21.15.5:0/2507589526 >> v2:172.21.15.5:6800/4269212764 conn(0x7fd81405c210 0x7fd81405e6d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fd82c104b00 tx=0x7fd81c009d60).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:22.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.543+0000 7fd81affd700 1 -- 172.21.15.5:0/2507589526 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7fd82404ab90 con 0x7fd82c0f6b10 2021-03-09T08:59:22.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:22.739+0000 7fd82b59e700 1 -- 172.21.15.5:0/2507589526 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd82c05f650 con 0x7fd81405c210 2021-03-09T08:59:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:22 smithi146 bash[13136]: cluster 2021-03-09T08:59:20.206606+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-09T08:59:23.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:22 smithi146 bash[13136]: cluster 2021-03-09T08:59:20.206795+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-09T08:59:23.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:22 smithi146 bash[13136]: cluster 2021-03-09T08:59:21.309509+0000 mgr.y (mgr.14144) 96 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:23.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:22 smithi146 bash[13136]: cluster 2021-03-09T08:59:21.959676+0000 mon.a (mon.0) 241 : cluster [INF] osd.1 v2:172.21.15.5:6805/137257932 boot 2021-03-09T08:59:23.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:22 smithi146 bash[13136]: cluster 2021-03-09T08:59:21.959740+0000 mon.a (mon.0) 242 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2021-03-09T08:59:23.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:22 smithi146 bash[13136]: audit 2021-03-09T08:59:21.960271+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-09T08:59:23.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:22 smithi146 bash[13136]: audit 2021-03-09T08:59:22.746215+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-09T08:59:23.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:22 smithi146 bash[13136]: audit 2021-03-09T08:59:22.748863+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-09T08:59:23.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:22 smithi146 bash[13136]: audit 2021-03-09T08:59:22.749828+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:59:23.277 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[12738]: cluster 2021-03-09T08:59:20.206606+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-09T08:59:23.277 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[12738]: cluster 2021-03-09T08:59:20.206795+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-09T08:59:23.277 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[12738]: cluster 2021-03-09T08:59:21.309509+0000 mgr.y (mgr.14144) 96 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:23.278 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[12738]: cluster 2021-03-09T08:59:21.959676+0000 mon.a (mon.0) 241 : cluster [INF] osd.1 v2:172.21.15.5:6805/137257932 boot 2021-03-09T08:59:23.278 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[12738]: cluster 2021-03-09T08:59:21.959740+0000 mon.a (mon.0) 242 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2021-03-09T08:59:23.278 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[12738]: audit 2021-03-09T08:59:21.960271+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-09T08:59:23.278 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[12738]: audit 2021-03-09T08:59:22.746215+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-09T08:59:23.279 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[12738]: audit 2021-03-09T08:59:22.748863+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-09T08:59:23.279 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[12738]: audit 2021-03-09T08:59:22.749828+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:59:23.279 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[16589]: cluster 2021-03-09T08:59:20.206606+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-09T08:59:23.280 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[16589]: cluster 2021-03-09T08:59:20.206795+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-09T08:59:23.280 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[16589]: cluster 2021-03-09T08:59:21.309509+0000 mgr.y (mgr.14144) 96 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-09T08:59:23.280 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[16589]: cluster 2021-03-09T08:59:21.959676+0000 mon.a (mon.0) 241 : cluster [INF] osd.1 v2:172.21.15.5:6805/137257932 boot 2021-03-09T08:59:23.280 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[16589]: cluster 2021-03-09T08:59:21.959740+0000 mon.a (mon.0) 242 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2021-03-09T08:59:23.280 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[16589]: audit 2021-03-09T08:59:21.960271+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-09T08:59:23.281 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[16589]: audit 2021-03-09T08:59:22.746215+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-09T08:59:23.281 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[16589]: audit 2021-03-09T08:59:22.748863+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-09T08:59:23.281 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:22 smithi005 bash[16589]: audit 2021-03-09T08:59:22.749828+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:59:24.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:23 smithi146 bash[13136]: audit 2021-03-09T08:59:22.744666+0000 mgr.y (mgr.14144) 97 : audit [DBG] from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:59:24.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:23 smithi146 bash[13136]: cephadm 2021-03-09T08:59:22.748504+0000 mgr.y (mgr.14144) 98 : cephadm [INF] Applying service osd.None on host smithi005... 2021-03-09T08:59:24.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:23 smithi146 bash[13136]: cluster 2021-03-09T08:59:22.974558+0000 mon.a (mon.0) 247 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-03-09T08:59:24.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:23 smithi005 bash[16589]: audit 2021-03-09T08:59:22.744666+0000 mgr.y (mgr.14144) 97 : audit [DBG] from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:59:24.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:23 smithi005 bash[16589]: cephadm 2021-03-09T08:59:22.748504+0000 mgr.y (mgr.14144) 98 : cephadm [INF] Applying service osd.None on host smithi005... 2021-03-09T08:59:24.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:23 smithi005 bash[16589]: cluster 2021-03-09T08:59:22.974558+0000 mon.a (mon.0) 247 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-03-09T08:59:24.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:23 smithi005 bash[12738]: audit 2021-03-09T08:59:22.744666+0000 mgr.y (mgr.14144) 97 : audit [DBG] from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-03-09T08:59:24.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:23 smithi005 bash[12738]: cephadm 2021-03-09T08:59:22.748504+0000 mgr.y (mgr.14144) 98 : cephadm [INF] Applying service osd.None on host smithi005... 2021-03-09T08:59:24.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:23 smithi005 bash[12738]: cluster 2021-03-09T08:59:22.974558+0000 mon.a (mon.0) 247 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-03-09T08:59:25.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:24 smithi005 bash[12738]: cluster 2021-03-09T08:59:23.309987+0000 mgr.y (mgr.14144) 99 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:25.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:24 smithi005 bash[12738]: audit 2021-03-09T08:59:24.707844+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:25.116 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:24 smithi005 bash[16589]: cluster 2021-03-09T08:59:23.309987+0000 mgr.y (mgr.14144) 99 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:25.117 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:24 smithi005 bash[16589]: audit 2021-03-09T08:59:24.707844+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:24 smithi146 bash[13136]: cluster 2021-03-09T08:59:23.309987+0000 mgr.y (mgr.14144) 99 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:24 smithi146 bash[13136]: audit 2021-03-09T08:59:24.707844+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:26.253 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:25 smithi005 bash[12738]: cephadm 2021-03-09T08:59:24.699730+0000 mgr.y (mgr.14144) 100 : cephadm [INF] Detected new or changed devices on smithi005 2021-03-09T08:59:26.253 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:25 smithi005 bash[12738]: cephadm 2021-03-09T08:59:24.708653+0000 mgr.y (mgr.14144) 101 : cephadm [INF] Refreshing smithi005 facts 2021-03-09T08:59:26.254 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:25 smithi005 bash[12738]: audit 2021-03-09T08:59:25.136781+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:26.254 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:25 smithi005 bash[12738]: audit 2021-03-09T08:59:25.139095+0000 mon.a (mon.0) 250 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:59:26.254 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:25 smithi005 bash[12738]: audit 2021-03-09T08:59:25.145303+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:26.255 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:25 smithi005 bash[16589]: cephadm 2021-03-09T08:59:24.699730+0000 mgr.y (mgr.14144) 100 : cephadm [INF] Detected new or changed devices on smithi005 2021-03-09T08:59:26.255 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:25 smithi005 bash[16589]: cephadm 2021-03-09T08:59:24.708653+0000 mgr.y (mgr.14144) 101 : cephadm [INF] Refreshing smithi005 facts 2021-03-09T08:59:26.255 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:25 smithi005 bash[16589]: audit 2021-03-09T08:59:25.136781+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:26.255 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:25 smithi005 bash[16589]: audit 2021-03-09T08:59:25.139095+0000 mon.a (mon.0) 250 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:59:26.255 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:25 smithi005 bash[16589]: audit 2021-03-09T08:59:25.145303+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:25 smithi146 bash[13136]: cephadm 2021-03-09T08:59:24.699730+0000 mgr.y (mgr.14144) 100 : cephadm [INF] Detected new or changed devices on smithi005 2021-03-09T08:59:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:26 smithi146 bash[13136]: cephadm 2021-03-09T08:59:24.708653+0000 mgr.y (mgr.14144) 101 : cephadm [INF] Refreshing smithi005 facts 2021-03-09T08:59:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:26 smithi146 bash[13136]: audit 2021-03-09T08:59:25.136781+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:26 smithi146 bash[13136]: audit 2021-03-09T08:59:25.139095+0000 mon.a (mon.0) 250 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:59:26.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:26 smithi146 bash[13136]: audit 2021-03-09T08:59:25.145303+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:27.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:26 smithi146 bash[13136]: cluster 2021-03-09T08:59:25.310481+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:27.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:26 smithi005 bash[12738]: cluster 2021-03-09T08:59:25.310481+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:27.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:26 smithi005 bash[16589]: cluster 2021-03-09T08:59:25.310481+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:28.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:27 smithi146 bash[13136]: audit 2021-03-09T08:59:27.275276+0000 mon.c (mon.1) 10 : audit [INF] from='client.? 172.21.15.5:0/4249775081' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be1a27a5-4e79-4b24-b2c0-f14621083b1f"}]: dispatch 2021-03-09T08:59:28.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:27 smithi146 bash[13136]: audit 2021-03-09T08:59:27.275965+0000 mon.a (mon.0) 252 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be1a27a5-4e79-4b24-b2c0-f14621083b1f"}]: dispatch 2021-03-09T08:59:28.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:27 smithi146 bash[13136]: audit 2021-03-09T08:59:27.281359+0000 mon.a (mon.0) 253 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be1a27a5-4e79-4b24-b2c0-f14621083b1f"}]': finished 2021-03-09T08:59:28.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:27 smithi146 bash[13136]: cluster 2021-03-09T08:59:27.281445+0000 mon.a (mon.0) 254 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-03-09T08:59:28.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:27 smithi146 bash[13136]: audit 2021-03-09T08:59:27.284070+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T08:59:28.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:27 smithi005 bash[12738]: audit 2021-03-09T08:59:27.275276+0000 mon.c (mon.1) 10 : audit [INF] from='client.? 172.21.15.5:0/4249775081' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be1a27a5-4e79-4b24-b2c0-f14621083b1f"}]: dispatch 2021-03-09T08:59:28.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:27 smithi005 bash[12738]: audit 2021-03-09T08:59:27.275965+0000 mon.a (mon.0) 252 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be1a27a5-4e79-4b24-b2c0-f14621083b1f"}]: dispatch 2021-03-09T08:59:28.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:27 smithi005 bash[12738]: audit 2021-03-09T08:59:27.281359+0000 mon.a (mon.0) 253 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be1a27a5-4e79-4b24-b2c0-f14621083b1f"}]': finished 2021-03-09T08:59:28.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:27 smithi005 bash[12738]: cluster 2021-03-09T08:59:27.281445+0000 mon.a (mon.0) 254 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-03-09T08:59:28.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:27 smithi005 bash[12738]: audit 2021-03-09T08:59:27.284070+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T08:59:28.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:27 smithi005 bash[16589]: audit 2021-03-09T08:59:27.275276+0000 mon.c (mon.1) 10 : audit [INF] from='client.? 172.21.15.5:0/4249775081' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be1a27a5-4e79-4b24-b2c0-f14621083b1f"}]: dispatch 2021-03-09T08:59:28.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:27 smithi005 bash[16589]: audit 2021-03-09T08:59:27.275965+0000 mon.a (mon.0) 252 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be1a27a5-4e79-4b24-b2c0-f14621083b1f"}]: dispatch 2021-03-09T08:59:28.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:27 smithi005 bash[16589]: audit 2021-03-09T08:59:27.281359+0000 mon.a (mon.0) 253 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be1a27a5-4e79-4b24-b2c0-f14621083b1f"}]': finished 2021-03-09T08:59:28.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:27 smithi005 bash[16589]: cluster 2021-03-09T08:59:27.281445+0000 mon.a (mon.0) 254 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-03-09T08:59:28.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:27 smithi005 bash[16589]: audit 2021-03-09T08:59:27.284070+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T08:59:29.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:28 smithi146 bash[13136]: cluster 2021-03-09T08:59:27.310958+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:29.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:28 smithi146 bash[13136]: audit 2021-03-09T08:59:28.086185+0000 mon.c (mon.1) 11 : audit [DBG] from='client.? 172.21.15.5:0/773304770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-09T08:59:29.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:28 smithi005 bash[12738]: cluster 2021-03-09T08:59:27.310958+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:29.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:28 smithi005 bash[12738]: audit 2021-03-09T08:59:28.086185+0000 mon.c (mon.1) 11 : audit [DBG] from='client.? 172.21.15.5:0/773304770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-09T08:59:29.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:28 smithi005 bash[16589]: cluster 2021-03-09T08:59:27.310958+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:29.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:28 smithi005 bash[16589]: audit 2021-03-09T08:59:28.086185+0000 mon.c (mon.1) 11 : audit [DBG] from='client.? 172.21.15.5:0/773304770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-09T08:59:31.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:30 smithi146 bash[13136]: cluster 2021-03-09T08:59:29.311487+0000 mgr.y (mgr.14144) 104 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:31.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:31 smithi005 bash[16589]: cluster 2021-03-09T08:59:29.311487+0000 mgr.y (mgr.14144) 104 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:31.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:30 smithi005 bash[12738]: cluster 2021-03-09T08:59:29.311487+0000 mgr.y (mgr.14144) 104 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:33.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:33 smithi146 bash[13136]: cluster 2021-03-09T08:59:31.312059+0000 mgr.y (mgr.14144) 105 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:33.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:33 smithi005 bash[12738]: cluster 2021-03-09T08:59:31.312059+0000 mgr.y (mgr.14144) 105 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:33.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:33 smithi005 bash[16589]: cluster 2021-03-09T08:59:31.312059+0000 mgr.y (mgr.14144) 105 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:35.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:35 smithi146 bash[13136]: cluster 2021-03-09T08:59:33.312572+0000 mgr.y (mgr.14144) 106 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:35.365 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:35 smithi005 bash[16589]: cluster 2021-03-09T08:59:33.312572+0000 mgr.y (mgr.14144) 106 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:35.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:35 smithi005 bash[12738]: cluster 2021-03-09T08:59:33.312572+0000 mgr.y (mgr.14144) 106 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:36.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:36 smithi005 bash[12738]: cluster 2021-03-09T08:59:35.313145+0000 mgr.y (mgr.14144) 107 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:36.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:36 smithi005 bash[12738]: audit 2021-03-09T08:59:35.765487+0000 mon.a (mon.0) 256 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-03-09T08:59:36.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:36 smithi005 bash[12738]: audit 2021-03-09T08:59:35.766993+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:59:36.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:36 smithi005 bash[12738]: cephadm 2021-03-09T08:59:35.768259+0000 mgr.y (mgr.14144) 108 : cephadm [INF] Deploying daemon osd.2 on smithi005 2021-03-09T08:59:36.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:36 smithi005 bash[16589]: cluster 2021-03-09T08:59:35.313145+0000 mgr.y (mgr.14144) 107 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:36.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:36 smithi005 bash[16589]: audit 2021-03-09T08:59:35.765487+0000 mon.a (mon.0) 256 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-03-09T08:59:36.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:36 smithi005 bash[16589]: audit 2021-03-09T08:59:35.766993+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:59:36.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:36 smithi005 bash[16589]: cephadm 2021-03-09T08:59:35.768259+0000 mgr.y (mgr.14144) 108 : cephadm [INF] Deploying daemon osd.2 on smithi005 2021-03-09T08:59:36.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:36 smithi146 bash[13136]: cluster 2021-03-09T08:59:35.313145+0000 mgr.y (mgr.14144) 107 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:36.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:36 smithi146 bash[13136]: audit 2021-03-09T08:59:35.765487+0000 mon.a (mon.0) 256 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-03-09T08:59:36.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:36 smithi146 bash[13136]: audit 2021-03-09T08:59:35.766993+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-09T08:59:36.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:36 smithi146 bash[13136]: cephadm 2021-03-09T08:59:35.768259+0000 mgr.y (mgr.14144) 108 : cephadm [INF] Deploying daemon osd.2 on smithi005 2021-03-09T08:59:38.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:38.575+0000 7fd81affd700 1 -- 172.21.15.5:0/2507589526 <== mgr.14144 v2:172.21.15.5:6800/4269212764 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd82c05f650 con 0x7fd81405c210 2021-03-09T08:59:38.575 INFO:teuthology.orchestra.run.smithi005.stdout:Created osd(s) 2 on host 'smithi005' 2021-03-09T08:59:38.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:38.575+0000 7fd82b59e700 1 -- 172.21.15.5:0/2507589526 >> v2:172.21.15.5:6800/4269212764 conn(0x7fd81405c210 msgr2=0x7fd81405e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:59:38.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:38.575+0000 7fd82b59e700 1 --2- 172.21.15.5:0/2507589526 >> v2:172.21.15.5:6800/4269212764 conn(0x7fd81405c210 0x7fd81405e6d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fd82c104b00 tx=0x7fd81c009d60).stop 2021-03-09T08:59:38.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:38.575+0000 7fd82b59e700 1 -- 172.21.15.5:0/2507589526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82c0f6b10 msgr2=0x7fd82c106090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:59:38.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:38.575+0000 7fd82b59e700 1 --2- 172.21.15.5:0/2507589526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82c0f6b10 0x7fd82c106090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fd824013750 tx=0x7fd824004d10).stop 2021-03-09T08:59:38.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:38.579+0000 7fd82b59e700 1 -- 172.21.15.5:0/2507589526 shutdown_connections 2021-03-09T08:59:38.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:38.579+0000 7fd82b59e700 1 --2- 172.21.15.5:0/2507589526 >> v2:172.21.15.5:6800/4269212764 conn(0x7fd81405c210 0x7fd81405e6d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:38.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:38.579+0000 7fd82b59e700 1 --2- 172.21.15.5:0/2507589526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82c0f6b10 0x7fd82c106090 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:38.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:38.579+0000 7fd82b59e700 1 --2- 172.21.15.5:0/2507589526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd82c0f4f20 0x7fd82c104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:38.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:38.579+0000 7fd82b59e700 1 --2- 172.21.15.5:0/2507589526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd82c0f61c0 0x7fd82c105280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:38.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:38.579+0000 7fd82b59e700 1 -- 172.21.15.5:0/2507589526 >> 172.21.15.5:0/2507589526 conn(0x7fd82c0f0700 msgr2=0x7fd82c0f29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:59:38.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:38.579+0000 7fd82b59e700 1 -- 172.21.15.5:0/2507589526 shutdown_connections 2021-03-09T08:59:38.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:38.579+0000 7fd82b59e700 1 -- 172.21.15.5:0/2507589526 wait complete. 2021-03-09T08:59:39.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:38 smithi005 bash[12738]: cluster 2021-03-09T08:59:37.313604+0000 mgr.y (mgr.14144) 109 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:39.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:38 smithi005 bash[12738]: audit 2021-03-09T08:59:38.557976+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:59:39.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:38 smithi005 bash[12738]: audit 2021-03-09T08:59:38.568196+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:39.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:38 smithi005 bash[12738]: audit 2021-03-09T08:59:38.575576+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:39.164 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:38 smithi005 bash[16589]: cluster 2021-03-09T08:59:37.313604+0000 mgr.y (mgr.14144) 109 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:39.165 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:38 smithi005 bash[16589]: audit 2021-03-09T08:59:38.557976+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:59:39.165 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:38 smithi005 bash[16589]: audit 2021-03-09T08:59:38.568196+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:39.165 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:38 smithi005 bash[16589]: audit 2021-03-09T08:59:38.575576+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:38 smithi146 bash[13136]: cluster 2021-03-09T08:59:37.313604+0000 mgr.y (mgr.14144) 109 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:38 smithi146 bash[13136]: audit 2021-03-09T08:59:38.557976+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-09T08:59:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:38 smithi146 bash[13136]: audit 2021-03-09T08:59:38.568196+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:38 smithi146 bash[13136]: audit 2021-03-09T08:59:38.575576+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:40.005 DEBUG:teuthology.orchestra.run.smithi005:osd.2> sudo journalctl -f -n 0 -u ceph-252abbe0-80b5-11eb-907b-001a4aab830c@osd.2.service 2021-03-09T08:59:40.014 INFO:tasks.cephadm:Deploying osd.3 on smithi005 with /dev/vg_nvme/lv_1... 2021-03-09T08:59:40.014 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2021-03-09T08:59:40.028 INFO:journalctl@ceph.osd.2.smithi005.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-09T08:59:41.037 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:40 smithi005 bash[12738]: cluster 2021-03-09T08:59:39.314141+0000 mgr.y (mgr.14144) 110 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:41.038 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:40 smithi005 bash[16589]: cluster 2021-03-09T08:59:39.314141+0000 mgr.y (mgr.14144) 110 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:41.038 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:40 smithi005 bash[23991]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-03-09T08:59:41.039 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:40 smithi005 bash[23991]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2021-03-09T08:59:41.039 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:40 smithi005 bash[23991]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2021-03-09T08:59:41.039 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:40 smithi005 bash[23991]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2021-03-09T08:59:41.039 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:40 smithi005 bash[23991]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2021-03-09T08:59:41.040 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:40 smithi005 bash[23991]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-03-09T08:59:41.040 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:40 smithi005 bash[23991]: --> ceph-volume lvm activate successful for osd ID: 2 2021-03-09T08:59:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:40 smithi146 bash[13136]: cluster 2021-03-09T08:59:39.314141+0000 mgr.y (mgr.14144) 110 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:43.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:42 smithi146 bash[13136]: cluster 2021-03-09T08:59:41.314656+0000 mgr.y (mgr.14144) 111 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:43.365 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:42 smithi005 bash[12738]: cluster 2021-03-09T08:59:41.314656+0000 mgr.y (mgr.14144) 111 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:43.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:42 smithi005 bash[16589]: cluster 2021-03-09T08:59:41.314656+0000 mgr.y (mgr.14144) 111 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:43.866 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 20 -- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 msgr2=0x5628c9d45400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2021-03-09T08:59:43.866 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 20 -- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 msgr2=0x5628c9d45400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=96 2021-03-09T08:59:43.866 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 20 --2- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 0x5628c9d45400 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x5628c9d59ef0 tx=0x5628d1f8cd80).handle_read_frame_preamble_main r=0 2021-03-09T08:59:43.867 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 20 --2- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 0x5628c9d45400 secure :-1 s=THROTTLE_MESSAGE pgs=97 cs=0 l=1 rev1=1 rx=0x5628c9d59ef0 tx=0x5628d1f8cd80).throttle_message 2021-03-09T08:59:43.867 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 20 --2- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 0x5628c9d45400 secure :-1 s=THROTTLE_BYTES pgs=97 cs=0 l=1 rev1=1 rx=0x5628c9d59ef0 tx=0x5628d1f8cd80).throttle_bytes 2021-03-09T08:59:43.867 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 20 --2- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 0x5628c9d45400 secure :-1 s=THROTTLE_DISPATCH_QUEUE pgs=97 cs=0 l=1 rev1=1 rx=0x5628c9d59ef0 tx=0x5628d1f8cd80).throttle_dispatch_queue 2021-03-09T08:59:43.867 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 20 --2- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 0x5628c9d45400 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rev1=1 rx=0x5628c9d59ef0 tx=0x5628d1f8cd80).read_frame_segment seg_idx=0 2021-03-09T08:59:43.867 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 20 --2- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 0x5628c9d45400 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rev1=1 rx=0x5628c9d59ef0 tx=0x5628d1f8cd80).read_frame_segment seg_idx=1 2021-03-09T08:59:43.868 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 20 -- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 msgr2=0x5628c9d45400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=208 2021-03-09T08:59:43.868 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 20 --2- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 0x5628c9d45400 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rev1=1 rx=0x5628c9d59ef0 tx=0x5628d1f8cd80).handle_read_frame_segment r=0 2021-03-09T08:59:43.868 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 20 -- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 msgr2=0x5628c9d45400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2021-03-09T08:59:43.868 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 20 --2- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 0x5628c9d45400 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rev1=1 rx=0x5628c9d59ef0 tx=0x5628d1f8cd80).handle_read_frame_epilogue_main r=0 2021-03-09T08:59:43.869 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 10 --2- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 0x5628c9d45400 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rev1=1 rx=0x5628c9d59ef0 tx=0x5628d1f8cd80).handle_read_frame_dispatch tag=17 2021-03-09T08:59:43.869 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 20 --2- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 0x5628c9d45400 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rev1=1 rx=0x5628c9d59ef0 tx=0x5628d1f8cd80).handle_message 2021-03-09T08:59:43.869 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 5 --2- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 0x5628c9d45400 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rev1=1 rx=0x5628c9d59ef0 tx=0x5628d1f8cd80).handle_message got 194 + 0 + 0 byte message. envelope type=18 src mon.0 off 0 2021-03-09T08:59:43.869 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 5 --2- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 0x5628c9d45400 secure :-1 s=READ_MESSAGE_COMPLETE pgs=97 cs=0 l=1 rev1=1 rx=0x5628c9d59ef0 tx=0x5628d1f8cd80).handle_message received message m=0x5628d1f5c4e0 seq=4 from=mon.0 type=18 auth_reply(proto 2 0 (0) Success) v1 2021-03-09T08:59:43.869 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 20 -- 172.21.15.5:0/822675342 queue 0x5628d1f5c4e0 prio 196 2021-03-09T08:59:43.870 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 20 --2- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 0x5628c9d45400 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x5628c9d59ef0 tx=0x5628d1f8cd80).read_frame 2021-03-09T08:59:43.870 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38309e700 20 -- 172.21.15.5:0/822675342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5628d1fd8c00 msgr2=0x5628c9d45400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=96 2021-03-09T08:59:43.870 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38209c700 1 -- 172.21.15.5:0/822675342 <== mon.0 v2:172.21.15.5:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5628d1f5c4e0 con 0x5628d1fd8c00 2021-03-09T08:59:43.870 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38209c700 10 -- 172.21.15.5:0/822675342 dispatch_throttle_release 194 to dispatch throttler 194/104857600 2021-03-09T08:59:43.870 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:43 smithi005 bash[24159]: debug 2021-03-09T08:59:43.599+0000 7ff38209c700 20 -- 172.21.15.5:0/822675342 done calling dispatch on 0x5628d1f5c4e0 2021-03-09T08:59:44.615 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:44 smithi005 bash[24159]: debug 2021-03-09T08:59:44.171+0000 7ff389b31080 -1 Falling back to public interface 2021-03-09T08:59:45.007 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:44 smithi005 bash[12738]: cluster 2021-03-09T08:59:43.315187+0000 mgr.y (mgr.14144) 112 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:45.008 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:44 smithi005 bash[12738]: audit 2021-03-09T08:59:43.758023+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:45.008 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:44 smithi005 bash[16589]: cluster 2021-03-09T08:59:43.315187+0000 mgr.y (mgr.14144) 112 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:45.009 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:44 smithi005 bash[16589]: audit 2021-03-09T08:59:43.758023+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:45.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:44 smithi146 bash[13136]: cluster 2021-03-09T08:59:43.315187+0000 mgr.y (mgr.14144) 112 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:45.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:44 smithi146 bash[13136]: audit 2021-03-09T08:59:43.758023+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' 2021-03-09T08:59:45.676 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping: /dev/vg_nvme/lv_1 2021-03-09T08:59:45.676 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2021-03-09T08:59:45.677 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10+0 records in 2021-03-09T08:59:45.677 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2021-03-09T08:59:45.677 INFO:teuthology.orchestra.run.smithi005.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0171113 s, 613 MB/s 2021-03-09T08:59:45.677 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping successful for: 2021-03-09T08:59:45.791 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:45 smithi005 bash[12738]: audit 2021-03-09T08:59:45.485133+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-09T08:59:46.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:45 smithi005 bash[12738]: audit 2021-03-09T08:59:45.488122+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-09T08:59:46.116 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:45 smithi005 bash[16589]: audit 2021-03-09T08:59:45.485133+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-09T08:59:46.116 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:45 smithi005 bash[16589]: audit 2021-03-09T08:59:45.488122+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-09T08:59:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:45 smithi146 bash[13136]: audit 2021-03-09T08:59:45.485133+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-09T08:59:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:45 smithi146 bash[13136]: audit 2021-03-09T08:59:45.488122+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-09T08:59:46.865 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:46 smithi005 bash[12738]: cluster 2021-03-09T08:59:45.315642+0000 mgr.y (mgr.14144) 113 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:46.866 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:46 smithi005 bash[16589]: cluster 2021-03-09T08:59:45.315642+0000 mgr.y (mgr.14144) 113 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:46.922 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:89fb622b10ca2e58fe0c913a8956377a886e2ab4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 252abbe0-80b5-11eb-907b-001a4aab830c -- ceph orch daemon add osd smithi005:vg_nvme/lv_1 2021-03-09T08:59:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:46 smithi146 bash[13136]: cluster 2021-03-09T08:59:45.315642+0000 mgr.y (mgr.14144) 113 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:47.615 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:47 smithi005 bash[24159]: debug 2021-03-09T08:59:47.179+0000 7ff389b31080 -1 osd.2 0 log_to_monitors {default=true} 2021-03-09T08:59:48.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:47 smithi146 bash[13136]: audit 2021-03-09T08:59:47.187955+0000 mon.a (mon.0) 264 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-03-09T08:59:48.274 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:47 smithi005 bash[12738]: audit 2021-03-09T08:59:47.187955+0000 mon.a (mon.0) 264 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-03-09T08:59:48.274 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:47 smithi005 bash[16589]: audit 2021-03-09T08:59:47.187955+0000 mon.a (mon.0) 264 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-03-09T08:59:49.115 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 09 08:59:48 smithi005 bash[24159]: debug 2021-03-09T08:59:48.927+0000 7ff380098700 -1 osd.2 0 waiting for initial osdmap 2021-03-09T08:59:49.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:48 smithi005 bash[12738]: cluster 2021-03-09T08:59:47.316157+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:49.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:48 smithi005 bash[12738]: audit 2021-03-09T08:59:47.916855+0000 mon.a (mon.0) 265 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-03-09T08:59:49.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:48 smithi005 bash[12738]: cluster 2021-03-09T08:59:47.916955+0000 mon.a (mon.0) 266 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-03-09T08:59:49.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:48 smithi005 bash[12738]: audit 2021-03-09T08:59:47.917178+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T08:59:49.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:48 smithi005 bash[12738]: audit 2021-03-09T08:59:47.917872+0000 mon.a (mon.0) 268 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2021-03-09T08:59:49.118 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:48 smithi005 bash[16589]: cluster 2021-03-09T08:59:47.316157+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:49.118 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:48 smithi005 bash[16589]: audit 2021-03-09T08:59:47.916855+0000 mon.a (mon.0) 265 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-03-09T08:59:49.118 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:48 smithi005 bash[16589]: cluster 2021-03-09T08:59:47.916955+0000 mon.a (mon.0) 266 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-03-09T08:59:49.118 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:48 smithi005 bash[16589]: audit 2021-03-09T08:59:47.917178+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T08:59:49.119 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:48 smithi005 bash[16589]: audit 2021-03-09T08:59:47.917872+0000 mon.a (mon.0) 268 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2021-03-09T08:59:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:48 smithi146 bash[13136]: cluster 2021-03-09T08:59:47.316157+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-09T08:59:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:48 smithi146 bash[13136]: audit 2021-03-09T08:59:47.916855+0000 mon.a (mon.0) 265 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-03-09T08:59:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:48 smithi146 bash[13136]: cluster 2021-03-09T08:59:47.916955+0000 mon.a (mon.0) 266 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-03-09T08:59:49.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:48 smithi146 bash[13136]: audit 2021-03-09T08:59:47.917178+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T08:59:49.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:48 smithi146 bash[13136]: audit 2021-03-09T08:59:47.917872+0000 mon.a (mon.0) 268 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2021-03-09T08:59:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:49 smithi146 bash[13136]: audit 2021-03-09T08:59:48.920210+0000 mon.a (mon.0) 269 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T08:59:50.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:49 smithi146 bash[13136]: cluster 2021-03-09T08:59:48.920524+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-09T08:59:50.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:49 smithi146 bash[13136]: audit 2021-03-09T08:59:48.921424+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T08:59:50.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 08:59:49 smithi146 bash[13136]: audit 2021-03-09T08:59:48.930506+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T08:59:50.365 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:49 smithi005 bash[12738]: audit 2021-03-09T08:59:48.920210+0000 mon.a (mon.0) 269 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T08:59:50.365 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:49 smithi005 bash[12738]: cluster 2021-03-09T08:59:48.920524+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-09T08:59:50.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:49 smithi005 bash[12738]: audit 2021-03-09T08:59:48.921424+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T08:59:50.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:49 smithi005 bash[12738]: audit 2021-03-09T08:59:48.930506+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T08:59:50.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:49 smithi005 bash[16589]: audit 2021-03-09T08:59:48.920210+0000 mon.a (mon.0) 269 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T08:59:50.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:49 smithi005 bash[16589]: cluster 2021-03-09T08:59:48.920524+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-09T08:59:50.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:49 smithi005 bash[16589]: audit 2021-03-09T08:59:48.921424+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T08:59:50.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 08:59:49 smithi005 bash[16589]: audit 2021-03-09T08:59:48.930506+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T08:59:50.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 -- 172.21.15.5:0/3098855126 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f834c0f5930 msgr2=0x7f834c0f5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:59:50.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 --2- 172.21.15.5:0/3098855126 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f834c0f5930 0x7f834c0f5d90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f8334004e30 tx=0x7f8334009b30).stop 2021-03-09T08:59:50.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 -- 172.21.15.5:0/3098855126 shutdown_connections 2021-03-09T08:59:50.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 --2- 172.21.15.5:0/3098855126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f834c0fcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:50.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 --2- 172.21.15.5:0/3098855126 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f834c0f4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:50.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 --2- 172.21.15.5:0/3098855126 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f834c0f5930 0x7f834c0f5d90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:50.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 -- 172.21.15.5:0/3098855126 >> 172.21.15.5:0/3098855126 conn(0x7f834c0efe90 msgr2=0x7f834c0f22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:59:50.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 -- 172.21.15.5:0/3098855126 shutdown_connections 2021-03-09T08:59:50.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 -- 172.21.15.5:0/3098855126 wait complete. 2021-03-09T08:59:50.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 Processor -- start 2021-03-09T08:59:50.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 -- start start 2021-03-09T08:59:50.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f834c108f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:50.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f834c10b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:50.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f834c0f6280 0x7f834c10d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:50.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834c10dec0 con 0x7f834c0f4690 2021-03-09T08:59:50.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834c10e000 con 0x7f834c0f5930 2021-03-09T08:59:50.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f834c10e1c0 con 0x7f834c0f6280 2021-03-09T08:59:50.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834affd700 1 -- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f834c108f40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T08:59:50.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834affd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f834c108f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:50.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834b7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f834c0f6280 0x7f834c10d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:50.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834b7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f834c0f6280 0x7f834c10d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:53234/0 (socket says 172.21.15.5:53234) 2021-03-09T08:59:50.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 learned_addr learned my addr 172.21.15.5:0/3543657960 (peer_addr_for_me v2:172.21.15.5:0/0) 2021-03-09T08:59:50.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f834c10b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:50.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f834c10b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:59:50.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f834c10b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:50.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f834c108f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:50.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f834c108f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:50.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834c10e380 con 0x7f834c0f6280 2021-03-09T08:59:50.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f834c0f6280 0x7f834c10d940 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f834400db70 tx=0x7f8344004e60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:50.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.1 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004020 con 0x7f834c0f6280 2021-03-09T08:59:50.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.1 v2:172.21.15.5:3301/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400f040 con 0x7f834c0f6280 2021-03-09T08:59:50.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.1 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440036c0 con 0x7f834c0f6280 2021-03-09T08:59:50.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f834c10e630 con 0x7f834c0f6280 2021-03-09T08:59:50.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.627+0000 7f834bfff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f834c10eb50 con 0x7f834c0f6280 2021-03-09T08:59:50.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.631+0000 7f834bfff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f834c0f3a80 con 0x7f834c0f6280 2021-03-09T08:59:50.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.631+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.1 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344003af0 con 0x7f834c0f6280 2021-03-09T08:59:50.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.631+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833805c1c0 0x7f833805e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:50.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.631+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833805c1c0 0x7f833805e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:50.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.631+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.1 v2:172.21.15.5:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 2068+0+0 (secure 0 0 0) 0x7f8344075030 con 0x7f834c0f6280 2021-03-09T08:59:50.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.631+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833805c1c0 0x7f833805e680 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f834c0ff600 tx=0x7f833c006ac0).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:50.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.635+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.1 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150121 (secure 0 0 0) 0x7f8344045a80 con 0x7f834c0f6280 2021-03-09T08:59:50.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T08:59:50.835+0000 7f834bfff700 1 -- 172.21.15.5:0/3543657960 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f834c0ffd30 con 0x7f833805c1c0 2021-03-09T08:59:50.866 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f132cea3700 time 2021-03-09T08:59:50.399290+0000 2021-03-09T08:59:50.867 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:59:50.867 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:59:50.867 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f133aa261d8] 2021-03-09T08:59:50.867 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f133aa263f2] 2021-03-09T08:59:50.868 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x556588fb397c] 2021-03-09T08:59:50.868 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x556588ff2ced] 2021-03-09T08:59:50.868 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55658901ab5a] 2021-03-09T08:59:50.869 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x556589026054] 2021-03-09T08:59:50.869 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55658902a2c3] 2021-03-09T08:59:50.869 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x556588fabc2d] 2021-03-09T08:59:50.869 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x556588e94c74] 2021-03-09T08:59:50.869 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x556588e994d9] 2021-03-09T08:59:50.870 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 11: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x556588e9a766] 2021-03-09T08:59:50.870 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 12: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x556588ec8b0c] 2021-03-09T08:59:50.870 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 13: (DispatchQueue::entry()+0x126a) [0x7f133ac600ea] 2021-03-09T08:59:50.870 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f133ad0dae1] 2021-03-09T08:59:50.870 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 15: (Thread::_entry_func(void*)+0xd) [0x7f133ab0d41d] 2021-03-09T08:59:50.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 16: /lib64/libpthread.so.0(+0x814a) [0x7f133850c14a] 2021-03-09T08:59:50.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 17: clone() 2021-03-09T08:59:50.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: *** Caught signal (Aborted) ** 2021-03-09T08:59:50.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: in thread 7f132cea3700 thread_name:ms_dispatch 2021-03-09T08:59:50.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: debug 2021-03-09T08:59:50.399+0000 7f132cea3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f132cea3700 time 2021-03-09T08:59:50.399290+0000 2021-03-09T08:59:50.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:59:50.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:59:50.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f133aa261d8] 2021-03-09T08:59:50.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f133aa263f2] 2021-03-09T08:59:50.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x556588fb397c] 2021-03-09T08:59:50.873 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x556588ff2ced] 2021-03-09T08:59:50.873 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55658901ab5a] 2021-03-09T08:59:50.873 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x556589026054] 2021-03-09T08:59:50.873 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55658902a2c3] 2021-03-09T08:59:50.874 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x556588fabc2d] 2021-03-09T08:59:50.874 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x556588e94c74] 2021-03-09T08:59:50.874 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x556588e994d9] 2021-03-09T08:59:50.875 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 11: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x556588e9a766] 2021-03-09T08:59:50.875 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 12: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x556588ec8b0c] 2021-03-09T08:59:50.875 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 13: (DispatchQueue::entry()+0x126a) [0x7f133ac600ea] 2021-03-09T08:59:50.876 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f133ad0dae1] 2021-03-09T08:59:50.876 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 15: (Thread::_entry_func(void*)+0xd) [0x7f133ab0d41d] 2021-03-09T08:59:50.877 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 16: /lib64/libpthread.so.0(+0x814a) [0x7f133850c14a] 2021-03-09T08:59:50.877 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 17: clone() 2021-03-09T08:59:50.877 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:59:50.878 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f1338516b20] 2021-03-09T08:59:50.878 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 2: gsignal() 2021-03-09T08:59:50.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 3: abort() 2021-03-09T08:59:50.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f133aa26229] 2021-03-09T08:59:50.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f133aa263f2] 2021-03-09T08:59:50.880 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x556588fb397c] 2021-03-09T08:59:50.880 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x556588ff2ced] 2021-03-09T08:59:50.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55658901ab5a] 2021-03-09T08:59:50.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x556589026054] 2021-03-09T08:59:50.882 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55658902a2c3] 2021-03-09T08:59:50.882 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x556588fabc2d] 2021-03-09T08:59:50.882 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x556588e94c74] 2021-03-09T08:59:50.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x556588e994d9] 2021-03-09T08:59:50.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x556588e9a766] 2021-03-09T08:59:50.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x556588ec8b0c] 2021-03-09T08:59:50.886 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 16: (DispatchQueue::entry()+0x126a) [0x7f133ac600ea] 2021-03-09T08:59:50.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f133ad0dae1] 2021-03-09T08:59:50.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 18: (Thread::_entry_func(void*)+0xd) [0x7f133ab0d41d] 2021-03-09T08:59:50.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 19: /lib64/libpthread.so.0(+0x814a) [0x7f133850c14a] 2021-03-09T08:59:50.888 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 20: clone() 2021-03-09T08:59:50.888 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: debug 2021-03-09T08:59:50.403+0000 7f132cea3700 -1 *** Caught signal (Aborted) ** 2021-03-09T08:59:50.889 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: in thread 7f132cea3700 thread_name:ms_dispatch 2021-03-09T08:59:50.889 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:59:50.889 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f1338516b20] 2021-03-09T08:59:50.890 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 2: gsignal() 2021-03-09T08:59:50.890 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 3: abort() 2021-03-09T08:59:50.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f133aa26229] 2021-03-09T08:59:50.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f133aa263f2] 2021-03-09T08:59:50.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x556588fb397c] 2021-03-09T08:59:50.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x556588ff2ced] 2021-03-09T08:59:50.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55658901ab5a] 2021-03-09T08:59:50.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x556589026054] 2021-03-09T08:59:50.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55658902a2c3] 2021-03-09T08:59:50.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x556588fabc2d] 2021-03-09T08:59:50.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x556588e94c74] 2021-03-09T08:59:50.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x556588e994d9] 2021-03-09T08:59:50.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x556588e9a766] 2021-03-09T08:59:50.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x556588ec8b0c] 2021-03-09T08:59:50.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 16: (DispatchQueue::entry()+0x126a) [0x7f133ac600ea] 2021-03-09T08:59:50.895 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f133ad0dae1] 2021-03-09T08:59:50.895 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 18: (Thread::_entry_func(void*)+0xd) [0x7f133ab0d41d] 2021-03-09T08:59:50.895 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 19: /lib64/libpthread.so.0(+0x814a) [0x7f133850c14a] 2021-03-09T08:59:50.895 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 20: clone() 2021-03-09T08:59:50.896 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T08:59:50.896 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: debug -1> 2021-03-09T08:59:50.399+0000 7f132cea3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f132cea3700 time 2021-03-09T08:59:50.399290+0000 2021-03-09T08:59:50.896 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:59:50.897 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:59:50.897 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f133aa261d8] 2021-03-09T08:59:50.897 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f133aa263f2] 2021-03-09T08:59:50.898 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x556588fb397c] 2021-03-09T08:59:50.898 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x556588ff2ced] 2021-03-09T08:59:50.898 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55658901ab5a] 2021-03-09T08:59:50.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x556589026054] 2021-03-09T08:59:50.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55658902a2c3] 2021-03-09T08:59:50.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x556588fabc2d] 2021-03-09T08:59:50.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x556588e94c74] 2021-03-09T08:59:50.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x556588e994d9] 2021-03-09T08:59:50.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 11: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x556588e9a766] 2021-03-09T08:59:50.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 12: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x556588ec8b0c] 2021-03-09T08:59:50.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 13: (DispatchQueue::entry()+0x126a) [0x7f133ac600ea] 2021-03-09T08:59:50.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f133ad0dae1] 2021-03-09T08:59:50.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 15: (Thread::_entry_func(void*)+0xd) [0x7f133ab0d41d] 2021-03-09T08:59:50.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 16: /lib64/libpthread.so.0(+0x814a) [0x7f133850c14a] 2021-03-09T08:59:50.903 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 17: clone() 2021-03-09T08:59:50.903 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: debug 0> 2021-03-09T08:59:50.403+0000 7f132cea3700 -1 *** Caught signal (Aborted) ** 2021-03-09T08:59:50.903 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: in thread 7f132cea3700 thread_name:ms_dispatch 2021-03-09T08:59:50.904 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:59:50.904 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f1338516b20] 2021-03-09T08:59:50.905 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 2: gsignal() 2021-03-09T08:59:50.905 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 3: abort() 2021-03-09T08:59:50.906 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f133aa26229] 2021-03-09T08:59:50.906 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f133aa263f2] 2021-03-09T08:59:50.906 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x556588fb397c] 2021-03-09T08:59:50.907 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x556588ff2ced] 2021-03-09T08:59:50.907 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55658901ab5a] 2021-03-09T08:59:50.908 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x556589026054] 2021-03-09T08:59:50.908 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55658902a2c3] 2021-03-09T08:59:50.909 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x556588fabc2d] 2021-03-09T08:59:50.909 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x556588e94c74] 2021-03-09T08:59:50.909 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x556588e994d9] 2021-03-09T08:59:50.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x556588e9a766] 2021-03-09T08:59:50.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x556588ec8b0c] 2021-03-09T08:59:50.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 16: (DispatchQueue::entry()+0x126a) [0x7f133ac600ea] 2021-03-09T08:59:50.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f133ad0dae1] 2021-03-09T08:59:50.911 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 18: (Thread::_entry_func(void*)+0xd) [0x7f133ab0d41d] 2021-03-09T08:59:50.911 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 19: /lib64/libpthread.so.0(+0x814a) [0x7f133850c14a] 2021-03-09T08:59:50.911 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 20: clone() 2021-03-09T08:59:50.911 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T08:59:50.911 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: debug -1> 2021-03-09T08:59:50.399+0000 7f132cea3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f132cea3700 time 2021-03-09T08:59:50.399290+0000 2021-03-09T08:59:50.912 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:59:50.912 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:59:50.912 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f133aa261d8] 2021-03-09T08:59:50.912 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f133aa263f2] 2021-03-09T08:59:50.912 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x556588fb397c] 2021-03-09T08:59:50.913 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x556588ff2ced] 2021-03-09T08:59:50.913 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55658901ab5a] 2021-03-09T08:59:50.913 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x556589026054] 2021-03-09T08:59:50.913 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55658902a2c3] 2021-03-09T08:59:50.914 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x556588fabc2d] 2021-03-09T08:59:50.914 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x556588e94c74] 2021-03-09T08:59:50.914 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x556588e994d9] 2021-03-09T08:59:50.914 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 11: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x556588e9a766] 2021-03-09T08:59:50.914 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 12: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x556588ec8b0c] 2021-03-09T08:59:50.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 13: (DispatchQueue::entry()+0x126a) [0x7f133ac600ea] 2021-03-09T08:59:50.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f133ad0dae1] 2021-03-09T08:59:50.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 15: (Thread::_entry_func(void*)+0xd) [0x7f133ab0d41d] 2021-03-09T08:59:50.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 16: /lib64/libpthread.so.0(+0x814a) [0x7f133850c14a] 2021-03-09T08:59:50.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 17: clone() 2021-03-09T08:59:50.916 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: debug 0> 2021-03-09T08:59:50.403+0000 7f132cea3700 -1 *** Caught signal (Aborted) ** 2021-03-09T08:59:50.916 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: in thread 7f132cea3700 thread_name:ms_dispatch 2021-03-09T08:59:50.916 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:59:50.916 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f1338516b20] 2021-03-09T08:59:50.916 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 2: gsignal() 2021-03-09T08:59:50.917 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 3: abort() 2021-03-09T08:59:50.917 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f133aa26229] 2021-03-09T08:59:50.917 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f133aa263f2] 2021-03-09T08:59:50.917 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x556588fb397c] 2021-03-09T08:59:50.917 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x556588ff2ced] 2021-03-09T08:59:50.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55658901ab5a] 2021-03-09T08:59:50.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x556589026054] 2021-03-09T08:59:50.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55658902a2c3] 2021-03-09T08:59:50.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x556588fabc2d] 2021-03-09T08:59:50.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x556588e94c74] 2021-03-09T08:59:50.919 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x556588e994d9] 2021-03-09T08:59:50.919 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x556588e9a766] 2021-03-09T08:59:50.919 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x556588ec8b0c] 2021-03-09T08:59:50.919 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 16: (DispatchQueue::entry()+0x126a) [0x7f133ac600ea] 2021-03-09T08:59:50.919 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f133ad0dae1] 2021-03-09T08:59:50.920 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 18: (Thread::_entry_func(void*)+0xd) [0x7f133ab0d41d] 2021-03-09T08:59:50.920 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 19: /lib64/libpthread.so.0(+0x814a) [0x7f133850c14a] 2021-03-09T08:59:50.920 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: 20: clone() 2021-03-09T08:59:50.920 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:50 smithi005 bash[12738]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T08:59:52.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:51 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Main process exited, code=exited, status=134/n/a 2021-03-09T08:59:52.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 08:59:51 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Failed with result 'exit-code'. 2021-03-09T09:00:02.365 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:01 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Scheduled restart job, restart counter is at 1. 2021-03-09T09:00:02.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:01 smithi005 systemd[1]: Stopped Ceph mon.a for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:00:02.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:01 smithi005 systemd[1]: Started Ceph mon.a for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:00:03.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.131+0000 7fc9357b3700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-09T09:00:03.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.131+0000 7fc9357b3700 0 ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev), process ceph-mon, pid 6 2021-03-09T09:00:03.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.131+0000 7fc9357b3700 0 pidfile_write: ignore empty --pid-file 2021-03-09T09:00:03.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 0 load: jerasure load: lrc load: isa 2021-03-09T09:00:03.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: RocksDB version: 6.8.1 2021-03-09T09:00:03.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-09T09:00:03.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Compile date Mar 9 2021 2021-03-09T09:00:03.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: DB SUMMARY 2021-03-09T09:00:03.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: CURRENT file: CURRENT 2021-03-09T09:00:03.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-09T09:00:03.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: MANIFEST file: MANIFEST-000008 size: 202 Bytes 2021-03-09T09:00:03.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 2, files: 000004.sst 000007.sst 2021-03-09T09:00:03.373 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 8235260 ; 2021-03-09T09:00:03.373 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.error_if_exists: 0 2021-03-09T09:00:03.373 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.create_if_missing: 0 2021-03-09T09:00:03.373 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.paranoid_checks: 1 2021-03-09T09:00:03.374 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.env: 0x55faf3c231c0 2021-03-09T09:00:03.374 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.fs: Posix File System 2021-03-09T09:00:03.374 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.info_log: 0x55fafd8d9120 2021-03-09T09:00:03.374 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-09T09:00:03.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.statistics: (nil) 2021-03-09T09:00:03.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.use_fsync: 0 2021-03-09T09:00:03.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_log_file_size: 0 2021-03-09T09:00:03.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-09T09:00:03.376 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-09T09:00:03.376 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-09T09:00:03.376 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-09T09:00:03.377 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.allow_fallocate: 1 2021-03-09T09:00:03.377 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-09T09:00:03.377 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-09T09:00:03.377 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.use_direct_reads: 0 2021-03-09T09:00:03.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-09T09:00:03.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-09T09:00:03.381 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.db_log_dir: 2021-03-09T09:00:03.381 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2021-03-09T09:00:03.381 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-09T09:00:03.381 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_subcompactions: 1 2021-03-09T09:00:03.382 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_background_flushes: -1 2021-03-09T09:00:03.382 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-09T09:00:03.382 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-09T09:00:03.382 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-09T09:00:03.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-09T09:00:03.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-09T09:00:03.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-09T09:00:03.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-09T09:00:03.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.write_buffer_manager: 0x55fafd92dcb0 2021-03-09T09:00:03.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-09T09:00:03.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-09T09:00:03.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-09T09:00:03.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-09T09:00:03.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-09T09:00:03.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-09T09:00:03.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-09T09:00:03.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-09T09:00:03.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-09T09:00:03.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.unordered_write: 0 2021-03-09T09:00:03.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-09T09:00:03.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-09T09:00:03.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-09T09:00:03.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-09T09:00:03.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.row_cache: None 2021-03-09T09:00:03.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.wal_filter: None 2021-03-09T09:00:03.389 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-09T09:00:03.389 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-09T09:00:03.389 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.preserve_deletes: 0 2021-03-09T09:00:03.389 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.two_write_queues: 0 2021-03-09T09:00:03.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-09T09:00:03.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.atomic_flush: 0 2021-03-09T09:00:03.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-09T09:00:03.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-09T09:00:03.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-09T09:00:03.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.log_readahead_size: 0 2021-03-09T09:00:03.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-09T09:00:03.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_background_jobs: 2 2021-03-09T09:00:03.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_background_compactions: -1 2021-03-09T09:00:03.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-09T09:00:03.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-09T09:00:03.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-09T09:00:03.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-09T09:00:03.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-09T09:00:03.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-09T09:00:03.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-09T09:00:03.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-09T09:00:03.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_open_files: -1 2021-03-09T09:00:03.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-09T09:00:03.396 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-09T09:00:03.396 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-09T09:00:03.396 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-09T09:00:03.396 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Compression algorithms supported: 2021-03-09T09:00:03.396 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-09T09:00:03.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: kZSTD supported: 0 2021-03-09T09:00:03.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: kXpressCompression supported: 0 2021-03-09T09:00:03.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-09T09:00:03.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: kLZ4Compression supported: 1 2021-03-09T09:00:03.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: kBZip2Compression supported: 0 2021-03-09T09:00:03.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: kZlibCompression supported: 1 2021-03-09T09:00:03.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: kSnappyCompression supported: 1 2021-03-09T09:00:03.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-09T09:00:03.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000008 2021-03-09T09:00:03.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-09T09:00:03.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-09T09:00:03.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.merge_operator: 2021-03-09T09:00:03.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compaction_filter: None 2021-03-09T09:00:03.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-09T09:00:03.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-09T09:00:03.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-09T09:00:03.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55faf567c230) 2021-03-09T09:00:03.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: cache_index_and_filter_blocks: 1 2021-03-09T09:00:03.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-09T09:00:03.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-09T09:00:03.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: pin_top_level_index_and_filter: 1 2021-03-09T09:00:03.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: index_type: 0 2021-03-09T09:00:03.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: data_block_index_type: 0 2021-03-09T09:00:03.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: index_shortening: 1 2021-03-09T09:00:03.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: data_block_hash_table_util_ratio: 0.750000 2021-03-09T09:00:03.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: hash_index_allow_collision: 1 2021-03-09T09:00:03.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: checksum: 1 2021-03-09T09:00:03.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: no_block_cache: 0 2021-03-09T09:00:03.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: block_cache: 0x55fafd8b4510 2021-03-09T09:00:03.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: block_cache_name: BinnedLRUCache 2021-03-09T09:00:03.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: block_cache_options: 2021-03-09T09:00:03.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: capacity : 536870912 2021-03-09T09:00:03.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: num_shard_bits : 4 2021-03-09T09:00:03.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: strict_capacity_limit : 0 2021-03-09T09:00:03.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: high_pri_pool_ratio: 0.000 2021-03-09T09:00:03.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: block_cache_compressed: (nil) 2021-03-09T09:00:03.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: persistent_cache: (nil) 2021-03-09T09:00:03.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: block_size: 4096 2021-03-09T09:00:03.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: block_size_deviation: 10 2021-03-09T09:00:03.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: block_restart_interval: 16 2021-03-09T09:00:03.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: index_block_restart_interval: 1 2021-03-09T09:00:03.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: metadata_block_size: 4096 2021-03-09T09:00:03.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: partition_filters: 0 2021-03-09T09:00:03.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: use_delta_encoding: 1 2021-03-09T09:00:03.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-09T09:00:03.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: whole_key_filtering: 1 2021-03-09T09:00:03.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: verify_compression: 0 2021-03-09T09:00:03.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: read_amp_bytes_per_bit: 0 2021-03-09T09:00:03.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: format_version: 2 2021-03-09T09:00:03.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: enable_index_compression: 1 2021-03-09T09:00:03.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: block_align: 0 2021-03-09T09:00:03.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-09T09:00:03.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-09T09:00:03.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compression: NoCompression 2021-03-09T09:00:03.408 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-09T09:00:03.408 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-09T09:00:03.408 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-09T09:00:03.408 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.num_levels: 7 2021-03-09T09:00:03.408 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-09T09:00:03.409 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-09T09:00:03.409 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-09T09:00:03.409 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-09T09:00:03.409 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-09T09:00:03.409 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-09T09:00:03.410 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-09T09:00:03.410 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:00:03.410 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-09T09:00:03.410 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-09T09:00:03.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-09T09:00:03.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-09T09:00:03.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-09T09:00:03.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:00:03.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-09T09:00:03.412 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-09T09:00:03.412 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-09T09:00:03.412 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-09T09:00:03.412 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-09T09:00:03.412 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-09T09:00:03.413 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-09T09:00:03.413 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-09T09:00:03.413 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-09T09:00:03.413 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-09T09:00:03.413 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-09T09:00:03.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-09T09:00:03.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-09T09:00:03.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-09T09:00:03.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-09T09:00:03.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-09T09:00:03.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-09T09:00:03.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-09T09:00:03.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-09T09:00:03.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-09T09:00:03.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-09T09:00:03.417 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-09T09:00:03.417 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-09T09:00:03.417 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-09T09:00:03.417 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-09T09:00:03.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-09T09:00:03.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-09T09:00:03.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-09T09:00:03.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-09T09:00:03.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-09T09:00:03.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-09T09:00:03.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-09T09:00:03.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-09T09:00:03.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.table_properties_collectors: 2021-03-09T09:00:03.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.inplace_update_support: 0 2021-03-09T09:00:03.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-09T09:00:03.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-09T09:00:03.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-09T09:00:03.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-09T09:00:03.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.bloom_locality: 0 2021-03-09T09:00:03.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.max_successive_merges: 0 2021-03-09T09:00:03.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-09T09:00:03.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-09T09:00:03.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-09T09:00:03.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-09T09:00:03.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.ttl: 2592000 2021-03-09T09:00:03.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.135+0000 7fc9357b3700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-09T09:00:03.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.139+0000 7fc9357b3700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 10, last_sequence is 247, log_number is 7,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-09T09:00:03.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.139+0000 7fc9357b3700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 7 2021-03-09T09:00:03.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.139+0000 7fc9357b3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280403141004, "job": 1, "event": "recovery_started", "log_files": [9]} 2021-03-09T09:00:03.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.139+0000 7fc9357b3700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #9 mode 2 2021-03-09T09:00:03.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.155+0000 7fc9357b3700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-09T09:00:03.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.167+0000 7fc9357b3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280403169354, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 10, "file_size": 7955866, "table_properties": {"data_size": 7939463, "index_size": 13210, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 2373, "raw_key_size": 17635, "raw_average_key_size": 18, "raw_value_size": 7916815, "raw_average_value_size": 8512, "num_data_blocks": 423, "num_entries": 930, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1615280403, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-09T09:00:03.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.167+0000 7fc9357b3700 4 rocksdb: [version_set.cc:3826] Creating manifest 11 2021-03-09T09:00:03.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.167+0000 7fc9357b3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280403170239, "job": 1, "event": "recovery_finished"} 2021-03-09T09:00:03.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.171+0000 7fc9357b3700 4 rocksdb: DB pointer 0x55fafd979800 2021-03-09T09:00:03.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.171+0000 7fc9247c3700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-09T09:00:03.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.171+0000 7fc9247c3700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-09T09:00:03.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: ** DB Stats ** 2021-03-09T09:00:03.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:00:03.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-09T09:00:03.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-09T09:00:03.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:00:03.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-09T09:00:03.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-09T09:00:03.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:00:03.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: ** Compaction Stats [default] ** 2021-03-09T09:00:03.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:03.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:03.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: L0 3/0 7.67 MB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 729.2 0.01 0.00 1 0.010 0 0 2021-03-09T09:00:03.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Sum 3/0 7.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 729.2 0.01 0.00 1 0.010 0 0 2021-03-09T09:00:03.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 729.2 0.01 0.00 1 0.010 0 0 2021-03-09T09:00:03.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: ** Compaction Stats [default] ** 2021-03-09T09:00:03.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:03.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:03.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 729.2 0.01 0.00 1 0.010 0 0 2021-03-09T09:00:03.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:00:03.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Flush(GB): cumulative 0.007, interval 0.007 2021-03-09T09:00:03.429 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:00:03.429 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:00:03.429 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:00:03.429 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:00:03.429 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Cumulative compaction: 0.01 GB write, 222.30 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:03.430 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Interval compaction: 0.01 GB write, 222.30 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:03.430 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:00:03.430 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:00:03.430 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: ** Compaction Stats [default] ** 2021-03-09T09:00:03.430 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:03.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:03.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: L0 3/0 7.67 MB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 729.2 0.01 0.00 1 0.010 0 0 2021-03-09T09:00:03.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Sum 3/0 7.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 729.2 0.01 0.00 1 0.010 0 0 2021-03-09T09:00:03.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:00:03.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: ** Compaction Stats [default] ** 2021-03-09T09:00:03.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:03.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:03.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 729.2 0.01 0.00 1 0.010 0 0 2021-03-09T09:00:03.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:00:03.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Flush(GB): cumulative 0.007, interval 0.000 2021-03-09T09:00:03.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:00:03.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:00:03.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:00:03.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:00:03.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Cumulative compaction: 0.01 GB write, 221.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:03.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:03.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:00:03.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:00:03.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.171+0000 7fc9357b3700 0 starting mon.a rank 0 at public addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] at bind addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:00:03.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.175+0000 7fc9357b3700 1 mon.a@-1(???) e3 preinit fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:00:03.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.175+0000 7fc9357b3700 0 mon.a@-1(???).mds e1 new map 2021-03-09T09:00:03.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.175+0000 7fc9357b3700 0 mon.a@-1(???).mds e1 print_map 2021-03-09T09:00:03.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: e1 2021-03-09T09:00:03.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: enable_multiple, ever_enabled_multiple: 1,1 2021-03-09T09:00:03.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-03-09T09:00:03.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: legacy client fscid: -1 2021-03-09T09:00:03.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: 2021-03-09T09:00:03.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: No filesystems configured 2021-03-09T09:00:03.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.175+0000 7fc9357b3700 0 mon.a@-1(???).osd e18 crush map has features 3314932999778484224, adjusting msgr requires 2021-03-09T09:00:03.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.175+0000 7fc9357b3700 0 mon.a@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:00:03.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.175+0000 7fc9357b3700 0 mon.a@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:00:03.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.175+0000 7fc9357b3700 0 mon.a@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:00:03.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: audit 2021-03-09T08:59:48.920210+0000 mon.a (mon.0) 269 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T09:00:03.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: cluster 2021-03-09T08:59:48.920524+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-09T09:00:03.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: audit 2021-03-09T08:59:48.921424+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:00:03.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: audit 2021-03-09T08:59:48.930506+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:00:03.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.175+0000 7fc9357b3700 1 mon.a@-1(???).paxosservice(auth 1..10) refresh upgraded, format 0 -> 3 2021-03-09T09:00:03.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.179+0000 7fc9357b3700 20 expand_channel_meta expand map: {default=false} 2021-03-09T09:00:03.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.179+0000 7fc9357b3700 20 expand_channel_meta from 'false' to 'false' 2021-03-09T09:00:03.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.179+0000 7fc9357b3700 20 expand_channel_meta expanded map: {default=false} 2021-03-09T09:00:03.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.179+0000 7fc9357b3700 20 expand_channel_meta expand map: {default=info} 2021-03-09T09:00:03.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.179+0000 7fc9357b3700 20 expand_channel_meta from 'info' to 'info' 2021-03-09T09:00:03.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.179+0000 7fc9357b3700 20 expand_channel_meta expanded map: {default=info} 2021-03-09T09:00:03.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.179+0000 7fc9357b3700 20 expand_channel_meta expand map: {default=daemon} 2021-03-09T09:00:03.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.179+0000 7fc9357b3700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-03-09T09:00:03.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.179+0000 7fc9357b3700 20 expand_channel_meta expanded map: {default=daemon} 2021-03-09T09:00:03.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.179+0000 7fc9357b3700 20 expand_channel_meta expand map: {default=debug} 2021-03-09T09:00:03.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.179+0000 7fc9357b3700 20 expand_channel_meta from 'debug' to 'debug' 2021-03-09T09:00:03.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:03 smithi005 bash[25305]: debug 2021-03-09T09:00:03.179+0000 7fc9357b3700 20 expand_channel_meta expanded map: {default=debug} 2021-03-09T09:00:05.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f834c0f6280 msgr2=0x7f834c10d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:00:05.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f834c0f6280 msgr2=0x7f834c10d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:05.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f834c0f6280 0x7f834c10d940 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f834400db70 tx=0x7f8344004e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:05.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f834c0f6280 0x7f834c10d940 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f834400db70 tx=0x7f8344004e60).stop 2021-03-09T09:00:05.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f834c0f6280 msgr2=0x7f834c10d940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:05.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f834c0f6280 0x7f834c10d940 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:05.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338062790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:05.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338065130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:05.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338067750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:05.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067c60 con 0x7f834c0f4690 2021-03-09T09:00:05.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067da0 con 0x7f834c0f5930 2021-03-09T09:00:05.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806c340 con 0x7f8338062ca0 2021-03-09T09:00:05.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338065130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:00:05.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338062790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:05.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338065130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:05.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338067750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:05.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338065130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:05.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338065130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:05.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338062790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:00:05.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338062790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:05.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806c500 con 0x7f834c0f5930 2021-03-09T09:00:05.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.491+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338067750 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f833c006eb0 tx=0x7f834c05dff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:05.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c011710 con 0x7f834c0f5930 2021-03-09T09:00:05.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834c0ffd30 con 0x7f834c0f5930 2021-03-09T09:00:05.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:00:05.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00f070 con 0x7f834c0f5930 2021-03-09T09:00:05.865 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f201b9a0700 time 2021-03-09T09:00:05.435570+0000 2021-03-09T09:00:05.866 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:05.866 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:05.866 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f20295231d8] 2021-03-09T09:00:05.867 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f20295233f2] 2021-03-09T09:00:05.867 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55adc846797c] 2021-03-09T09:00:05.867 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55adc84a6ced] 2021-03-09T09:00:05.868 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55adc84ceb5a] 2021-03-09T09:00:05.868 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55adc84da054] 2021-03-09T09:00:05.868 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55adc84de2c3] 2021-03-09T09:00:05.868 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55adc845fc2d] 2021-03-09T09:00:05.868 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55adc83b40b4] 2021-03-09T09:00:05.869 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 10: (C_MonOp::finish(int)+0x49) [0x55adc8353069] 2021-03-09T09:00:05.869 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 11: (Context::complete(int)+0xd) [0x55adc835049d] 2021-03-09T09:00:05.869 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55adc837bda5] 2021-03-09T09:00:05.870 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 13: (Paxos::finish_round()+0x27b) [0x55adc845753b] 2021-03-09T09:00:05.872 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 14: (Paxos::handle_last(boost::intrusive_ptr)+0xe29) [0x55adc84587f9] 2021-03-09T09:00:05.872 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 15: (Paxos::dispatch(boost::intrusive_ptr)+0x417) [0x55adc8459497] 2021-03-09T09:00:05.872 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0xfc0) [0x55adc834dca0] 2021-03-09T09:00:05.872 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 17: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x55adc834e766] 2021-03-09T09:00:05.873 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x55adc837cb0c] 2021-03-09T09:00:05.873 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 19: (DispatchQueue::entry()+0x126a) [0x7f202975d0ea] 2021-03-09T09:00:05.873 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f202980aae1] 2021-03-09T09:00:05.874 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 21: (Thread::_entry_func(void*)+0xd) [0x7f202960a41d] 2021-03-09T09:00:05.874 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 22: /lib64/libpthread.so.0(+0x814a) [0x7f202700914a] 2021-03-09T09:00:05.874 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 23: clone() 2021-03-09T09:00:05.874 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: *** Caught signal (Aborted) ** 2021-03-09T09:00:05.875 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: in thread 7f201b9a0700 thread_name:ms_dispatch 2021-03-09T09:00:05.875 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: debug 2021-03-09T09:00:05.439+0000 7f201b9a0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f201b9a0700 time 2021-03-09T09:00:05.435570+0000 2021-03-09T09:00:05.876 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:05.876 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:05.876 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f20295231d8] 2021-03-09T09:00:05.876 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f20295233f2] 2021-03-09T09:00:05.876 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55adc846797c] 2021-03-09T09:00:05.877 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55adc84a6ced] 2021-03-09T09:00:05.877 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55adc84ceb5a] 2021-03-09T09:00:05.877 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55adc84da054] 2021-03-09T09:00:05.877 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55adc84de2c3] 2021-03-09T09:00:05.878 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55adc845fc2d] 2021-03-09T09:00:05.878 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55adc83b40b4] 2021-03-09T09:00:05.878 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 10: (C_MonOp::finish(int)+0x49) [0x55adc8353069] 2021-03-09T09:00:05.878 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 11: (Context::complete(int)+0xd) [0x55adc835049d] 2021-03-09T09:00:05.879 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55adc837bda5] 2021-03-09T09:00:05.879 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 13: (Paxos::finish_round()+0x27b) [0x55adc845753b] 2021-03-09T09:00:05.879 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 14: (Paxos::handle_last(boost::intrusive_ptr)+0xe29) [0x55adc84587f9] 2021-03-09T09:00:05.879 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 15: (Paxos::dispatch(boost::intrusive_ptr)+0x417) [0x55adc8459497] 2021-03-09T09:00:05.879 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0xfc0) [0x55adc834dca0] 2021-03-09T09:00:05.880 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 17: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x55adc834e766] 2021-03-09T09:00:05.880 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x55adc837cb0c] 2021-03-09T09:00:05.880 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 19: (DispatchQueue::entry()+0x126a) [0x7f202975d0ea] 2021-03-09T09:00:05.880 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f202980aae1] 2021-03-09T09:00:05.881 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 21: (Thread::_entry_func(void*)+0xd) [0x7f202960a41d] 2021-03-09T09:00:05.881 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 22: /lib64/libpthread.so.0(+0x814a) [0x7f202700914a] 2021-03-09T09:00:05.881 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 23: clone() 2021-03-09T09:00:05.881 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:05.882 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f2027013b20] 2021-03-09T09:00:05.882 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 2: gsignal() 2021-03-09T09:00:05.882 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 3: abort() 2021-03-09T09:00:05.883 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f2029523229] 2021-03-09T09:00:05.883 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f20295233f2] 2021-03-09T09:00:05.883 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55adc846797c] 2021-03-09T09:00:05.883 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55adc84a6ced] 2021-03-09T09:00:05.883 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55adc84ceb5a] 2021-03-09T09:00:05.884 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55adc84da054] 2021-03-09T09:00:05.884 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55adc84de2c3] 2021-03-09T09:00:05.884 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55adc845fc2d] 2021-03-09T09:00:05.884 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55adc83b40b4] 2021-03-09T09:00:05.884 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 13: (C_MonOp::finish(int)+0x49) [0x55adc8353069] 2021-03-09T09:00:05.885 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 14: (Context::complete(int)+0xd) [0x55adc835049d] 2021-03-09T09:00:05.885 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55adc837bda5] 2021-03-09T09:00:05.885 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 16: (Paxos::finish_round()+0x27b) [0x55adc845753b] 2021-03-09T09:00:05.885 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 17: (Paxos::handle_last(boost::intrusive_ptr)+0xe29) [0x55adc84587f9] 2021-03-09T09:00:05.886 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 18: (Paxos::dispatch(boost::intrusive_ptr)+0x417) [0x55adc8459497] 2021-03-09T09:00:05.886 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 19: (Monitor::dispatch_op(boost::intrusive_ptr)+0xfc0) [0x55adc834dca0] 2021-03-09T09:00:05.886 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 20: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x55adc834e766] 2021-03-09T09:00:05.887 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x55adc837cb0c] 2021-03-09T09:00:05.887 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 22: (DispatchQueue::entry()+0x126a) [0x7f202975d0ea] 2021-03-09T09:00:05.887 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 23: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f202980aae1] 2021-03-09T09:00:05.888 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 24: (Thread::_entry_func(void*)+0xd) [0x7f202960a41d] 2021-03-09T09:00:05.888 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 25: /lib64/libpthread.so.0(+0x814a) [0x7f202700914a] 2021-03-09T09:00:05.888 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 26: clone() 2021-03-09T09:00:05.888 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: debug 2021-03-09T09:00:05.447+0000 7f201b9a0700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:00:05.888 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: in thread 7f201b9a0700 thread_name:ms_dispatch 2021-03-09T09:00:05.889 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:05.889 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f2027013b20] 2021-03-09T09:00:05.889 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 2: gsignal() 2021-03-09T09:00:05.889 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 3: abort() 2021-03-09T09:00:05.889 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f2029523229] 2021-03-09T09:00:05.890 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f20295233f2] 2021-03-09T09:00:05.890 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55adc846797c] 2021-03-09T09:00:05.890 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55adc84a6ced] 2021-03-09T09:00:05.890 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55adc84ceb5a] 2021-03-09T09:00:05.891 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55adc84da054] 2021-03-09T09:00:05.891 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55adc84de2c3] 2021-03-09T09:00:05.891 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55adc845fc2d] 2021-03-09T09:00:05.892 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55adc83b40b4] 2021-03-09T09:00:05.892 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 13: (C_MonOp::finish(int)+0x49) [0x55adc8353069] 2021-03-09T09:00:05.892 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 14: (Context::complete(int)+0xd) [0x55adc835049d] 2021-03-09T09:00:05.892 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55adc837bda5] 2021-03-09T09:00:05.893 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 16: (Paxos::finish_round()+0x27b) [0x55adc845753b] 2021-03-09T09:00:05.893 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 17: (Paxos::handle_last(boost::intrusive_ptr)+0xe29) [0x55adc84587f9] 2021-03-09T09:00:05.893 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 18: (Paxos::dispatch(boost::intrusive_ptr)+0x417) [0x55adc8459497] 2021-03-09T09:00:05.893 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 19: (Monitor::dispatch_op(boost::intrusive_ptr)+0xfc0) [0x55adc834dca0] 2021-03-09T09:00:05.893 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 20: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x55adc834e766] 2021-03-09T09:00:05.894 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x55adc837cb0c] 2021-03-09T09:00:05.894 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 22: (DispatchQueue::entry()+0x126a) [0x7f202975d0ea] 2021-03-09T09:00:05.894 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 23: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f202980aae1] 2021-03-09T09:00:05.894 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 24: (Thread::_entry_func(void*)+0xd) [0x7f202960a41d] 2021-03-09T09:00:05.895 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 25: /lib64/libpthread.so.0(+0x814a) [0x7f202700914a] 2021-03-09T09:00:05.895 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 26: clone() 2021-03-09T09:00:05.895 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:00:05.895 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: debug -1> 2021-03-09T09:00:05.439+0000 7f201b9a0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f201b9a0700 time 2021-03-09T09:00:05.435570+0000 2021-03-09T09:00:05.895 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:05.896 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:05.896 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f20295231d8] 2021-03-09T09:00:05.896 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f20295233f2] 2021-03-09T09:00:05.896 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55adc846797c] 2021-03-09T09:00:05.896 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55adc84a6ced] 2021-03-09T09:00:05.897 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55adc84ceb5a] 2021-03-09T09:00:05.897 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55adc84da054] 2021-03-09T09:00:05.897 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55adc84de2c3] 2021-03-09T09:00:05.897 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55adc845fc2d] 2021-03-09T09:00:05.898 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55adc83b40b4] 2021-03-09T09:00:05.898 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 10: (C_MonOp::finish(int)+0x49) [0x55adc8353069] 2021-03-09T09:00:05.898 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 11: (Context::complete(int)+0xd) [0x55adc835049d] 2021-03-09T09:00:05.898 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55adc837bda5] 2021-03-09T09:00:05.899 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 13: (Paxos::finish_round()+0x27b) [0x55adc845753b] 2021-03-09T09:00:05.899 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 14: (Paxos::handle_last(boost::intrusive_ptr)+0xe29) [0x55adc84587f9] 2021-03-09T09:00:05.899 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 15: (Paxos::dispatch(boost::intrusive_ptr)+0x417) [0x55adc8459497] 2021-03-09T09:00:05.899 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0xfc0) [0x55adc834dca0] 2021-03-09T09:00:05.899 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 17: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x55adc834e766] 2021-03-09T09:00:05.900 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x55adc837cb0c] 2021-03-09T09:00:05.900 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 19: (DispatchQueue::entry()+0x126a) [0x7f202975d0ea] 2021-03-09T09:00:05.900 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f202980aae1] 2021-03-09T09:00:05.900 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 21: (Thread::_entry_func(void*)+0xd) [0x7f202960a41d] 2021-03-09T09:00:05.901 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 22: /lib64/libpthread.so.0(+0x814a) [0x7f202700914a] 2021-03-09T09:00:05.901 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 23: clone() 2021-03-09T09:00:05.901 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: debug 0> 2021-03-09T09:00:05.447+0000 7f201b9a0700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:00:05.901 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: in thread 7f201b9a0700 thread_name:ms_dispatch 2021-03-09T09:00:05.902 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:05.902 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f2027013b20] 2021-03-09T09:00:05.903 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 2: gsignal() 2021-03-09T09:00:05.903 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 3: abort() 2021-03-09T09:00:05.903 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f2029523229] 2021-03-09T09:00:05.903 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f20295233f2] 2021-03-09T09:00:05.903 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55adc846797c] 2021-03-09T09:00:05.904 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55adc84a6ced] 2021-03-09T09:00:05.904 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55adc84ceb5a] 2021-03-09T09:00:05.904 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55adc84da054] 2021-03-09T09:00:05.904 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55adc84de2c3] 2021-03-09T09:00:05.904 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55adc845fc2d] 2021-03-09T09:00:05.905 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55adc83b40b4] 2021-03-09T09:00:05.905 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 13: (C_MonOp::finish(int)+0x49) [0x55adc8353069] 2021-03-09T09:00:05.905 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 14: (Context::complete(int)+0xd) [0x55adc835049d] 2021-03-09T09:00:05.905 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55adc837bda5] 2021-03-09T09:00:05.905 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 16: (Paxos::finish_round()+0x27b) [0x55adc845753b] 2021-03-09T09:00:05.906 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 17: (Paxos::handle_last(boost::intrusive_ptr)+0xe29) [0x55adc84587f9] 2021-03-09T09:00:05.906 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 18: (Paxos::dispatch(boost::intrusive_ptr)+0x417) [0x55adc8459497] 2021-03-09T09:00:05.906 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 19: (Monitor::dispatch_op(boost::intrusive_ptr)+0xfc0) [0x55adc834dca0] 2021-03-09T09:00:05.906 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 20: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x55adc834e766] 2021-03-09T09:00:05.907 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x55adc837cb0c] 2021-03-09T09:00:05.907 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 22: (DispatchQueue::entry()+0x126a) [0x7f202975d0ea] 2021-03-09T09:00:05.907 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 23: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f202980aae1] 2021-03-09T09:00:05.907 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 24: (Thread::_entry_func(void*)+0xd) [0x7f202960a41d] 2021-03-09T09:00:05.907 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 25: /lib64/libpthread.so.0(+0x814a) [0x7f202700914a] 2021-03-09T09:00:05.908 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 26: clone() 2021-03-09T09:00:05.908 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:00:05.908 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: debug -1> 2021-03-09T09:00:05.439+0000 7f201b9a0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f201b9a0700 time 2021-03-09T09:00:05.435570+0000 2021-03-09T09:00:05.908 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:05.908 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:05.909 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f20295231d8] 2021-03-09T09:00:05.909 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f20295233f2] 2021-03-09T09:00:05.909 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55adc846797c] 2021-03-09T09:00:05.909 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55adc84a6ced] 2021-03-09T09:00:05.909 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55adc84ceb5a] 2021-03-09T09:00:05.910 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55adc84da054] 2021-03-09T09:00:05.910 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55adc84de2c3] 2021-03-09T09:00:05.910 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55adc845fc2d] 2021-03-09T09:00:05.911 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55adc83b40b4] 2021-03-09T09:00:05.911 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 10: (C_MonOp::finish(int)+0x49) [0x55adc8353069] 2021-03-09T09:00:05.911 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 11: (Context::complete(int)+0xd) [0x55adc835049d] 2021-03-09T09:00:05.911 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55adc837bda5] 2021-03-09T09:00:05.912 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 13: (Paxos::finish_round()+0x27b) [0x55adc845753b] 2021-03-09T09:00:05.912 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 14: (Paxos::handle_last(boost::intrusive_ptr)+0xe29) [0x55adc84587f9] 2021-03-09T09:00:05.912 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 15: (Paxos::dispatch(boost::intrusive_ptr)+0x417) [0x55adc8459497] 2021-03-09T09:00:05.912 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0xfc0) [0x55adc834dca0] 2021-03-09T09:00:05.913 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 17: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x55adc834e766] 2021-03-09T09:00:05.913 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x55adc837cb0c] 2021-03-09T09:00:05.913 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 19: (DispatchQueue::entry()+0x126a) [0x7f202975d0ea] 2021-03-09T09:00:05.914 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f202980aae1] 2021-03-09T09:00:05.914 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 21: (Thread::_entry_func(void*)+0xd) [0x7f202960a41d] 2021-03-09T09:00:05.914 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 22: /lib64/libpthread.so.0(+0x814a) [0x7f202700914a] 2021-03-09T09:00:05.914 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 23: clone() 2021-03-09T09:00:05.914 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: debug 0> 2021-03-09T09:00:05.447+0000 7f201b9a0700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:00:05.915 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: in thread 7f201b9a0700 thread_name:ms_dispatch 2021-03-09T09:00:05.915 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:05.915 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f2027013b20] 2021-03-09T09:00:05.915 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 2: gsignal() 2021-03-09T09:00:05.915 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 3: abort() 2021-03-09T09:00:05.916 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f2029523229] 2021-03-09T09:00:05.916 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f20295233f2] 2021-03-09T09:00:05.916 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55adc846797c] 2021-03-09T09:00:05.916 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55adc84a6ced] 2021-03-09T09:00:05.916 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55adc84ceb5a] 2021-03-09T09:00:05.917 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55adc84da054] 2021-03-09T09:00:05.917 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55adc84de2c3] 2021-03-09T09:00:05.917 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55adc845fc2d] 2021-03-09T09:00:05.917 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55adc83b40b4] 2021-03-09T09:00:05.917 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 13: (C_MonOp::finish(int)+0x49) [0x55adc8353069] 2021-03-09T09:00:05.918 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 14: (Context::complete(int)+0xd) [0x55adc835049d] 2021-03-09T09:00:05.918 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55adc837bda5] 2021-03-09T09:00:05.918 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 16: (Paxos::finish_round()+0x27b) [0x55adc845753b] 2021-03-09T09:00:05.918 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 17: (Paxos::handle_last(boost::intrusive_ptr)+0xe29) [0x55adc84587f9] 2021-03-09T09:00:05.919 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 18: (Paxos::dispatch(boost::intrusive_ptr)+0x417) [0x55adc8459497] 2021-03-09T09:00:05.919 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 19: (Monitor::dispatch_op(boost::intrusive_ptr)+0xfc0) [0x55adc834dca0] 2021-03-09T09:00:05.919 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 20: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x55adc834e766] 2021-03-09T09:00:05.919 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x55adc837cb0c] 2021-03-09T09:00:05.920 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 22: (DispatchQueue::entry()+0x126a) [0x7f202975d0ea] 2021-03-09T09:00:05.920 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 23: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f202980aae1] 2021-03-09T09:00:05.920 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 24: (Thread::_entry_func(void*)+0xd) [0x7f202960a41d] 2021-03-09T09:00:05.920 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 25: /lib64/libpthread.so.0(+0x814a) [0x7f202700914a] 2021-03-09T09:00:05.921 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: 26: clone() 2021-03-09T09:00:05.921 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:05 smithi005 bash[16589]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:00:06.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:06 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Main process exited, code=exited, status=134/n/a 2021-03-09T09:00:06.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:06 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Failed with result 'exit-code'. 2021-03-09T09:00:16.865 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:16 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Scheduled restart job, restart counter is at 1. 2021-03-09T09:00:16.866 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:16 smithi005 systemd[1]: Stopped Ceph mon.c for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:00:16.866 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:16 smithi005 systemd[1]: Started Ceph mon.c for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:00:17.866 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.575+0000 7fc9cb1d1700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-09T09:00:17.867 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.575+0000 7fc9cb1d1700 0 ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev), process ceph-mon, pid 7 2021-03-09T09:00:17.867 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.575+0000 7fc9cb1d1700 0 pidfile_write: ignore empty --pid-file 2021-03-09T09:00:17.867 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 0 load: jerasure load: lrc load: isa 2021-03-09T09:00:17.868 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: RocksDB version: 6.8.1 2021-03-09T09:00:17.868 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-09T09:00:17.869 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Compile date Mar 9 2021 2021-03-09T09:00:17.869 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: DB SUMMARY 2021-03-09T09:00:17.869 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: CURRENT file: CURRENT 2021-03-09T09:00:17.870 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-09T09:00:17.870 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2021-03-09T09:00:17.870 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000004.sst 2021-03-09T09:00:17.870 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000006.log size: 10616764 ; 2021-03-09T09:00:17.871 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.error_if_exists: 0 2021-03-09T09:00:17.871 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.create_if_missing: 0 2021-03-09T09:00:17.871 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.paranoid_checks: 1 2021-03-09T09:00:17.871 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.env: 0x555f5cc1e1c0 2021-03-09T09:00:17.872 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.fs: Posix File System 2021-03-09T09:00:17.872 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.info_log: 0x555f65e39120 2021-03-09T09:00:17.872 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-09T09:00:17.873 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.statistics: (nil) 2021-03-09T09:00:17.874 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.use_fsync: 0 2021-03-09T09:00:17.874 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_log_file_size: 0 2021-03-09T09:00:17.874 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-09T09:00:17.875 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-09T09:00:17.876 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-09T09:00:17.876 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-09T09:00:17.876 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.allow_fallocate: 1 2021-03-09T09:00:17.877 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-09T09:00:17.877 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-09T09:00:17.878 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.use_direct_reads: 0 2021-03-09T09:00:17.878 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-09T09:00:17.879 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-09T09:00:17.879 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.db_log_dir: 2021-03-09T09:00:17.880 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2021-03-09T09:00:17.880 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-09T09:00:17.881 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_subcompactions: 1 2021-03-09T09:00:17.881 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_background_flushes: -1 2021-03-09T09:00:17.882 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-09T09:00:17.883 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-09T09:00:17.883 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-09T09:00:17.884 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-09T09:00:17.884 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-09T09:00:17.884 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-09T09:00:17.885 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-09T09:00:17.885 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.write_buffer_manager: 0x555f65e8dce0 2021-03-09T09:00:17.886 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-09T09:00:17.886 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-09T09:00:17.887 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-09T09:00:17.887 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-09T09:00:17.888 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-09T09:00:17.888 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-09T09:00:17.888 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-09T09:00:17.889 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-09T09:00:17.893 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-09T09:00:17.893 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.unordered_write: 0 2021-03-09T09:00:17.894 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-09T09:00:17.894 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-09T09:00:17.895 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-09T09:00:17.895 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-09T09:00:17.896 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.row_cache: None 2021-03-09T09:00:17.896 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.wal_filter: None 2021-03-09T09:00:17.896 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-09T09:00:17.897 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-09T09:00:17.897 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.preserve_deletes: 0 2021-03-09T09:00:17.897 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.two_write_queues: 0 2021-03-09T09:00:17.897 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-09T09:00:17.897 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.atomic_flush: 0 2021-03-09T09:00:17.898 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-09T09:00:17.899 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-09T09:00:17.899 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-09T09:00:17.900 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.log_readahead_size: 0 2021-03-09T09:00:17.900 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-09T09:00:17.901 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_background_jobs: 2 2021-03-09T09:00:17.901 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_background_compactions: -1 2021-03-09T09:00:17.901 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-09T09:00:17.902 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-09T09:00:17.902 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-09T09:00:17.903 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-09T09:00:17.903 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-09T09:00:17.904 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-09T09:00:17.904 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-09T09:00:17.904 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-09T09:00:17.905 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_open_files: -1 2021-03-09T09:00:17.905 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-09T09:00:17.906 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-09T09:00:17.906 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-09T09:00:17.906 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-09T09:00:17.907 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Compression algorithms supported: 2021-03-09T09:00:17.907 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-09T09:00:17.908 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: kZSTD supported: 0 2021-03-09T09:00:17.908 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: kXpressCompression supported: 0 2021-03-09T09:00:17.909 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-09T09:00:17.909 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: kLZ4Compression supported: 1 2021-03-09T09:00:17.909 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: kBZip2Compression supported: 0 2021-03-09T09:00:17.910 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: kZlibCompression supported: 1 2021-03-09T09:00:17.910 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: kSnappyCompression supported: 1 2021-03-09T09:00:17.911 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-09T09:00:17.911 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2021-03-09T09:00:17.912 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-09T09:00:17.912 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-09T09:00:17.912 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.merge_operator: 2021-03-09T09:00:17.913 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compaction_filter: None 2021-03-09T09:00:17.913 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-09T09:00:17.914 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-09T09:00:17.914 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-09T09:00:17.915 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555f5dbdc230) 2021-03-09T09:00:17.915 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: cache_index_and_filter_blocks: 1 2021-03-09T09:00:17.916 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-09T09:00:17.916 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-09T09:00:17.916 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: pin_top_level_index_and_filter: 1 2021-03-09T09:00:17.917 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: index_type: 0 2021-03-09T09:00:17.917 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: data_block_index_type: 0 2021-03-09T09:00:17.918 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: index_shortening: 1 2021-03-09T09:00:17.918 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: data_block_hash_table_util_ratio: 0.750000 2021-03-09T09:00:17.919 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: hash_index_allow_collision: 1 2021-03-09T09:00:17.919 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: checksum: 1 2021-03-09T09:00:17.919 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: no_block_cache: 0 2021-03-09T09:00:17.920 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: block_cache: 0x555f65e14510 2021-03-09T09:00:17.920 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: block_cache_name: BinnedLRUCache 2021-03-09T09:00:17.921 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: block_cache_options: 2021-03-09T09:00:17.921 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: capacity : 536870912 2021-03-09T09:00:17.922 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: num_shard_bits : 4 2021-03-09T09:00:17.922 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: strict_capacity_limit : 0 2021-03-09T09:00:17.922 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: high_pri_pool_ratio: 0.000 2021-03-09T09:00:17.923 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: block_cache_compressed: (nil) 2021-03-09T09:00:17.923 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: persistent_cache: (nil) 2021-03-09T09:00:17.924 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: block_size: 4096 2021-03-09T09:00:17.924 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: block_size_deviation: 10 2021-03-09T09:00:17.925 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: block_restart_interval: 16 2021-03-09T09:00:17.925 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: index_block_restart_interval: 1 2021-03-09T09:00:17.925 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: metadata_block_size: 4096 2021-03-09T09:00:17.926 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: partition_filters: 0 2021-03-09T09:00:17.926 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: use_delta_encoding: 1 2021-03-09T09:00:17.927 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-09T09:00:17.927 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: whole_key_filtering: 1 2021-03-09T09:00:17.928 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: verify_compression: 0 2021-03-09T09:00:17.928 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: read_amp_bytes_per_bit: 0 2021-03-09T09:00:17.928 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: format_version: 2 2021-03-09T09:00:17.929 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: enable_index_compression: 1 2021-03-09T09:00:17.929 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: block_align: 0 2021-03-09T09:00:17.930 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-09T09:00:17.930 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-09T09:00:17.930 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compression: NoCompression 2021-03-09T09:00:17.931 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-09T09:00:17.931 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-09T09:00:17.931 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-09T09:00:17.931 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.num_levels: 7 2021-03-09T09:00:17.932 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-09T09:00:17.932 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-09T09:00:17.933 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-09T09:00:17.933 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-09T09:00:17.933 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-09T09:00:17.933 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-09T09:00:17.933 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-09T09:00:17.934 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:00:17.934 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-09T09:00:17.935 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-09T09:00:17.935 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-09T09:00:17.935 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-09T09:00:17.935 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-09T09:00:17.935 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:00:17.936 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-09T09:00:17.936 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-09T09:00:17.936 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-09T09:00:17.936 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-09T09:00:17.937 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-09T09:00:17.937 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-09T09:00:17.937 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-09T09:00:17.937 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-09T09:00:17.937 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-09T09:00:17.938 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-09T09:00:17.938 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-09T09:00:17.938 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-09T09:00:17.938 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-09T09:00:17.939 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-09T09:00:17.939 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-09T09:00:17.939 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-09T09:00:17.939 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-09T09:00:17.939 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-09T09:00:17.940 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-09T09:00:17.940 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-09T09:00:17.940 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-09T09:00:17.940 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-09T09:00:17.941 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-09T09:00:17.941 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-09T09:00:17.941 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-09T09:00:17.941 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-09T09:00:17.942 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-09T09:00:17.942 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-09T09:00:17.942 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-09T09:00:17.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-09T09:00:17.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-09T09:00:17.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-09T09:00:17.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-09T09:00:17.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.table_properties_collectors: 2021-03-09T09:00:17.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.inplace_update_support: 0 2021-03-09T09:00:17.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-09T09:00:17.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-09T09:00:17.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-09T09:00:17.945 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-09T09:00:17.945 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.bloom_locality: 0 2021-03-09T09:00:17.945 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.max_successive_merges: 0 2021-03-09T09:00:17.945 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-09T09:00:17.945 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-09T09:00:17.946 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-09T09:00:17.946 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-09T09:00:17.946 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.ttl: 2592000 2021-03-09T09:00:17.946 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-09T09:00:17.947 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-09T09:00:17.947 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2021-03-09T09:00:17.947 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280417583035, "job": 1, "event": "recovery_started", "log_files": [6]} 2021-03-09T09:00:17.947 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.579+0000 7fc9cb1d1700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2021-03-09T09:00:17.947 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.591+0000 7fc9cb1d1700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-09T09:00:17.948 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.603+0000 7fc9cb1d1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280417605144, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 8030718, "table_properties": {"data_size": 8013540, "index_size": 13473, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 2885, "raw_key_size": 26370, "raw_average_key_size": 23, "raw_value_size": 7986183, "raw_average_value_size": 7214, "num_data_blocks": 431, "num_entries": 1107, "num_deletions": 5, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1615280417, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-09T09:00:17.948 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.603+0000 7fc9cb1d1700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2021-03-09T09:00:17.948 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.607+0000 7fc9cb1d1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280417609951, "job": 1, "event": "recovery_finished"} 2021-03-09T09:00:17.948 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.611+0000 7fc9cb1d1700 4 rocksdb: DB pointer 0x555f65ed9800 2021-03-09T09:00:17.948 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.611+0000 7fc9ba1e1700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-09T09:00:17.949 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.611+0000 7fc9ba1e1700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-09T09:00:17.949 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: ** DB Stats ** 2021-03-09T09:00:17.949 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:00:17.949 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-09T09:00:17.949 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-09T09:00:17.950 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:00:17.950 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-09T09:00:17.950 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-09T09:00:17.950 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:00:17.951 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: ** Compaction Stats [default] ** 2021-03-09T09:00:17.951 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:17.951 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:17.951 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: L0 2/0 7.66 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 792.0 0.01 0.00 1 0.010 0 0 2021-03-09T09:00:17.951 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Sum 2/0 7.66 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 792.0 0.01 0.00 1 0.010 0 0 2021-03-09T09:00:17.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 792.0 0.01 0.00 1 0.010 0 0 2021-03-09T09:00:17.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: ** Compaction Stats [default] ** 2021-03-09T09:00:17.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:17.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:17.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 792.0 0.01 0.00 1 0.010 0 0 2021-03-09T09:00:17.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:00:17.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Flush(GB): cumulative 0.007, interval 0.007 2021-03-09T09:00:17.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:00:17.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:00:17.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:00:17.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:00:17.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Cumulative compaction: 0.01 GB write, 248.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:17.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Interval compaction: 0.01 GB write, 248.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:17.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:00:17.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:00:17.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: ** Compaction Stats [default] ** 2021-03-09T09:00:17.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:17.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:17.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: L0 2/0 7.66 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 792.0 0.01 0.00 1 0.010 0 0 2021-03-09T09:00:17.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Sum 2/0 7.66 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 792.0 0.01 0.00 1 0.010 0 0 2021-03-09T09:00:17.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:00:17.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: ** Compaction Stats [default] ** 2021-03-09T09:00:17.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:17.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:17.957 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 792.0 0.01 0.00 1 0.010 0 0 2021-03-09T09:00:17.957 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:00:17.957 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Flush(GB): cumulative 0.007, interval 0.000 2021-03-09T09:00:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:00:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:00:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:00:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:00:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Cumulative compaction: 0.01 GB write, 247.38 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:00:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:00:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.611+0000 7fc9cb1d1700 0 starting mon.c rank 1 at public addrs [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] at bind addrs [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:00:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.611+0000 7fc9cb1d1700 1 mon.c@-1(???) e3 preinit fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:00:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.615+0000 7fc9cb1d1700 0 mon.c@-1(???).mds e1 new map 2021-03-09T09:00:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.615+0000 7fc9cb1d1700 0 mon.c@-1(???).mds e1 print_map 2021-03-09T09:00:17.961 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: e1 2021-03-09T09:00:17.961 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: enable_multiple, ever_enabled_multiple: 1,1 2021-03-09T09:00:17.961 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-03-09T09:00:17.961 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: legacy client fscid: -1 2021-03-09T09:00:17.961 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: 2021-03-09T09:00:17.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: No filesystems configured 2021-03-09T09:00:17.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.615+0000 7fc9cb1d1700 0 mon.c@-1(???).osd e18 crush map has features 3314932999778484224, adjusting msgr requires 2021-03-09T09:00:17.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.615+0000 7fc9cb1d1700 0 mon.c@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:00:17.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.615+0000 7fc9cb1d1700 0 mon.c@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:00:17.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.615+0000 7fc9cb1d1700 0 mon.c@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:00:17.963 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: audit 2021-03-09T08:59:48.920210+0000 mon.a (mon.0) 269 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T09:00:17.963 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: cluster 2021-03-09T08:59:48.920524+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-09T09:00:17.963 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: audit 2021-03-09T08:59:48.921424+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:00:17.964 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: audit 2021-03-09T08:59:48.930506+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:00:17.964 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.615+0000 7fc9cb1d1700 1 mon.c@-1(???).paxosservice(auth 1..10) refresh upgraded, format 0 -> 3 2021-03-09T09:00:17.964 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.619+0000 7fc9cb1d1700 20 expand_channel_meta expand map: {default=false} 2021-03-09T09:00:17.964 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.619+0000 7fc9cb1d1700 20 expand_channel_meta from 'false' to 'false' 2021-03-09T09:00:17.964 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.619+0000 7fc9cb1d1700 20 expand_channel_meta expanded map: {default=false} 2021-03-09T09:00:17.965 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.619+0000 7fc9cb1d1700 20 expand_channel_meta expand map: {default=info} 2021-03-09T09:00:17.965 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.619+0000 7fc9cb1d1700 20 expand_channel_meta from 'info' to 'info' 2021-03-09T09:00:17.965 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.619+0000 7fc9cb1d1700 20 expand_channel_meta expanded map: {default=info} 2021-03-09T09:00:17.965 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.619+0000 7fc9cb1d1700 20 expand_channel_meta expand map: {default=daemon} 2021-03-09T09:00:17.965 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.619+0000 7fc9cb1d1700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-03-09T09:00:17.966 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.619+0000 7fc9cb1d1700 20 expand_channel_meta expanded map: {default=daemon} 2021-03-09T09:00:17.966 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.619+0000 7fc9cb1d1700 20 expand_channel_meta expand map: {default=debug} 2021-03-09T09:00:17.966 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.619+0000 7fc9cb1d1700 20 expand_channel_meta from 'debug' to 'debug' 2021-03-09T09:00:17.966 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:17 smithi005 bash[25433]: debug 2021-03-09T09:00:17.619+0000 7fc9cb1d1700 20 expand_channel_meta expanded map: {default=debug} 2021-03-09T09:00:24.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:00:24 smithi146 bash[13136]: debug 2021-03-09T09:00:24.354+0000 7fa560337700 -1 mon.b@2(electing) e3 get_health_metrics reporting 6 slow ops, oldest is mgrbeacon mgr.x(252abbe0-80b5-11eb-907b-001a4aab830c,34112, , 0) 2021-03-09T09:00:28.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fc9bc9e6700 time 2021-03-09T09:00:28.310983+0000 2021-03-09T09:00:28.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:28.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:28.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fc9ca5691d8] 2021-03-09T09:00:28.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fc9ca5693f2] 2021-03-09T09:00:28.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x555f5c48097c] 2021-03-09T09:00:28.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x555f5c4bfced] 2021-03-09T09:00:28.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x555f5c4e7b5a] 2021-03-09T09:00:28.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x555f5c4f3054] 2021-03-09T09:00:28.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x555f5c4f72c3] 2021-03-09T09:00:28.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x555f5c478c2d] 2021-03-09T09:00:28.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x555f5c361c74] 2021-03-09T09:00:28.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x555f5c3664d9] 2021-03-09T09:00:28.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 11: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x555f5c367766] 2021-03-09T09:00:28.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 12: (Monitor::handle_forward(boost::intrusive_ptr)+0x941) [0x555f5c368f91] 2021-03-09T09:00:28.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1266) [0x555f5c366f46] 2021-03-09T09:00:28.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x555f5c367766] 2021-03-09T09:00:28.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x555f5c395b0c] 2021-03-09T09:00:28.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 16: (DispatchQueue::entry()+0x126a) [0x7fc9ca7a30ea] 2021-03-09T09:00:28.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc9ca850ae1] 2021-03-09T09:00:28.621 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 18: (Thread::_entry_func(void*)+0xd) [0x7fc9ca65041d] 2021-03-09T09:00:28.621 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 19: /lib64/libpthread.so.0(+0x814a) [0x7fc9c804f14a] 2021-03-09T09:00:28.621 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 20: clone() 2021-03-09T09:00:28.621 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: debug 2021-03-09T09:00:28.315+0000 7fc9bc9e6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fc9bc9e6700 time 2021-03-09T09:00:28.310983+0000 2021-03-09T09:00:28.622 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:28.622 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:28.622 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fc9ca5691d8] 2021-03-09T09:00:28.623 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fc9ca5693f2] 2021-03-09T09:00:28.623 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x555f5c48097c] 2021-03-09T09:00:28.623 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x555f5c4bfced] 2021-03-09T09:00:28.623 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x555f5c4e7b5a] 2021-03-09T09:00:28.624 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x555f5c4f3054] 2021-03-09T09:00:28.624 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x555f5c4f72c3] 2021-03-09T09:00:28.625 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x555f5c478c2d] 2021-03-09T09:00:28.626 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x555f5c361c74] 2021-03-09T09:00:28.626 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x555f5c3664d9] 2021-03-09T09:00:28.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 11: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x555f5c367766] 2021-03-09T09:00:28.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 12: (Monitor::handle_forward(boost::intrusive_ptr)+0x941) [0x555f5c368f91] 2021-03-09T09:00:28.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1266) [0x555f5c366f46] 2021-03-09T09:00:28.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x555f5c367766] 2021-03-09T09:00:28.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x555f5c395b0c] 2021-03-09T09:00:28.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 16: (DispatchQueue::entry()+0x126a) [0x7fc9ca7a30ea] 2021-03-09T09:00:28.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc9ca850ae1] 2021-03-09T09:00:28.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 18: (Thread::_entry_func(void*)+0xd) [0x7fc9ca65041d] 2021-03-09T09:00:28.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 19: /lib64/libpthread.so.0(+0x814a) [0x7fc9c804f14a] 2021-03-09T09:00:28.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 20: clone() 2021-03-09T09:00:28.629 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: *** Caught signal (Aborted) ** 2021-03-09T09:00:28.629 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: in thread 7fc9bc9e6700 thread_name:ms_dispatch 2021-03-09T09:00:28.629 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:28.629 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc9c8059b20] 2021-03-09T09:00:28.629 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 2: gsignal() 2021-03-09T09:00:28.630 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 3: abort() 2021-03-09T09:00:28.630 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fc9ca569229] 2021-03-09T09:00:28.630 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fc9ca5693f2] 2021-03-09T09:00:28.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x555f5c48097c] 2021-03-09T09:00:28.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x555f5c4bfced] 2021-03-09T09:00:28.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x555f5c4e7b5a] 2021-03-09T09:00:28.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x555f5c4f3054] 2021-03-09T09:00:28.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x555f5c4f72c3] 2021-03-09T09:00:28.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x555f5c478c2d] 2021-03-09T09:00:28.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x555f5c361c74] 2021-03-09T09:00:28.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x555f5c3664d9] 2021-03-09T09:00:28.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x555f5c367766] 2021-03-09T09:00:28.633 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 15: (Monitor::handle_forward(boost::intrusive_ptr)+0x941) [0x555f5c368f91] 2021-03-09T09:00:28.633 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1266) [0x555f5c366f46] 2021-03-09T09:00:28.633 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 17: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x555f5c367766] 2021-03-09T09:00:28.633 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x555f5c395b0c] 2021-03-09T09:00:28.634 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 19: (DispatchQueue::entry()+0x126a) [0x7fc9ca7a30ea] 2021-03-09T09:00:28.634 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc9ca850ae1] 2021-03-09T09:00:28.634 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc9ca65041d] 2021-03-09T09:00:28.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 22: /lib64/libpthread.so.0(+0x814a) [0x7fc9c804f14a] 2021-03-09T09:00:28.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 23: clone() 2021-03-09T09:00:28.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: debug 2021-03-09T09:00:28.319+0000 7fc9bc9e6700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:00:28.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: in thread 7fc9bc9e6700 thread_name:ms_dispatch 2021-03-09T09:00:28.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:28.636 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc9c8059b20] 2021-03-09T09:00:28.636 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 2: gsignal() 2021-03-09T09:00:28.636 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 3: abort() 2021-03-09T09:00:28.636 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fc9ca569229] 2021-03-09T09:00:28.637 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fc9ca5693f2] 2021-03-09T09:00:28.637 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x555f5c48097c] 2021-03-09T09:00:28.637 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x555f5c4bfced] 2021-03-09T09:00:28.637 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x555f5c4e7b5a] 2021-03-09T09:00:28.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x555f5c4f3054] 2021-03-09T09:00:28.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x555f5c4f72c3] 2021-03-09T09:00:28.639 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x555f5c478c2d] 2021-03-09T09:00:28.639 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x555f5c361c74] 2021-03-09T09:00:28.639 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x555f5c3664d9] 2021-03-09T09:00:28.639 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x555f5c367766] 2021-03-09T09:00:28.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 15: (Monitor::handle_forward(boost::intrusive_ptr)+0x941) [0x555f5c368f91] 2021-03-09T09:00:28.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1266) [0x555f5c366f46] 2021-03-09T09:00:28.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 17: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x555f5c367766] 2021-03-09T09:00:28.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x555f5c395b0c] 2021-03-09T09:00:28.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 19: (DispatchQueue::entry()+0x126a) [0x7fc9ca7a30ea] 2021-03-09T09:00:28.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc9ca850ae1] 2021-03-09T09:00:28.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc9ca65041d] 2021-03-09T09:00:28.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 22: /lib64/libpthread.so.0(+0x814a) [0x7fc9c804f14a] 2021-03-09T09:00:28.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 23: clone() 2021-03-09T09:00:28.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:00:28.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: debug -1> 2021-03-09T09:00:28.315+0000 7fc9bc9e6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fc9bc9e6700 time 2021-03-09T09:00:28.310983+0000 2021-03-09T09:00:28.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:28.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:28.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fc9ca5691d8] 2021-03-09T09:00:28.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fc9ca5693f2] 2021-03-09T09:00:28.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x555f5c48097c] 2021-03-09T09:00:28.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x555f5c4bfced] 2021-03-09T09:00:28.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x555f5c4e7b5a] 2021-03-09T09:00:28.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x555f5c4f3054] 2021-03-09T09:00:28.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x555f5c4f72c3] 2021-03-09T09:00:28.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x555f5c478c2d] 2021-03-09T09:00:28.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x555f5c361c74] 2021-03-09T09:00:28.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x555f5c3664d9] 2021-03-09T09:00:28.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 11: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x555f5c367766] 2021-03-09T09:00:28.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 12: (Monitor::handle_forward(boost::intrusive_ptr)+0x941) [0x555f5c368f91] 2021-03-09T09:00:28.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1266) [0x555f5c366f46] 2021-03-09T09:00:28.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x555f5c367766] 2021-03-09T09:00:28.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x555f5c395b0c] 2021-03-09T09:00:28.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 16: (DispatchQueue::entry()+0x126a) [0x7fc9ca7a30ea] 2021-03-09T09:00:28.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc9ca850ae1] 2021-03-09T09:00:28.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 18: (Thread::_entry_func(void*)+0xd) [0x7fc9ca65041d] 2021-03-09T09:00:28.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 19: /lib64/libpthread.so.0(+0x814a) [0x7fc9c804f14a] 2021-03-09T09:00:28.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 20: clone() 2021-03-09T09:00:28.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: debug 0> 2021-03-09T09:00:28.319+0000 7fc9bc9e6700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:00:28.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: in thread 7fc9bc9e6700 thread_name:ms_dispatch 2021-03-09T09:00:28.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:28.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc9c8059b20] 2021-03-09T09:00:28.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 2: gsignal() 2021-03-09T09:00:28.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 3: abort() 2021-03-09T09:00:28.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fc9ca569229] 2021-03-09T09:00:28.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fc9ca5693f2] 2021-03-09T09:00:28.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x555f5c48097c] 2021-03-09T09:00:28.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x555f5c4bfced] 2021-03-09T09:00:28.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x555f5c4e7b5a] 2021-03-09T09:00:28.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x555f5c4f3054] 2021-03-09T09:00:28.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x555f5c4f72c3] 2021-03-09T09:00:28.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x555f5c478c2d] 2021-03-09T09:00:28.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x555f5c361c74] 2021-03-09T09:00:28.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x555f5c3664d9] 2021-03-09T09:00:28.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x555f5c367766] 2021-03-09T09:00:28.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 15: (Monitor::handle_forward(boost::intrusive_ptr)+0x941) [0x555f5c368f91] 2021-03-09T09:00:28.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1266) [0x555f5c366f46] 2021-03-09T09:00:28.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 17: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x555f5c367766] 2021-03-09T09:00:28.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x555f5c395b0c] 2021-03-09T09:00:28.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 19: (DispatchQueue::entry()+0x126a) [0x7fc9ca7a30ea] 2021-03-09T09:00:28.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc9ca850ae1] 2021-03-09T09:00:28.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc9ca65041d] 2021-03-09T09:00:28.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 22: /lib64/libpthread.so.0(+0x814a) [0x7fc9c804f14a] 2021-03-09T09:00:28.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 23: clone() 2021-03-09T09:00:28.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:00:28.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: debug -1> 2021-03-09T09:00:28.315+0000 7fc9bc9e6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fc9bc9e6700 time 2021-03-09T09:00:28.310983+0000 2021-03-09T09:00:28.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:28.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:28.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fc9ca5691d8] 2021-03-09T09:00:28.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fc9ca5693f2] 2021-03-09T09:00:28.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x555f5c48097c] 2021-03-09T09:00:28.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x555f5c4bfced] 2021-03-09T09:00:28.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x555f5c4e7b5a] 2021-03-09T09:00:28.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x555f5c4f3054] 2021-03-09T09:00:28.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x555f5c4f72c3] 2021-03-09T09:00:28.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x555f5c478c2d] 2021-03-09T09:00:28.658 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x555f5c361c74] 2021-03-09T09:00:28.658 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x555f5c3664d9] 2021-03-09T09:00:28.658 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 11: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x555f5c367766] 2021-03-09T09:00:28.658 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 12: (Monitor::handle_forward(boost::intrusive_ptr)+0x941) [0x555f5c368f91] 2021-03-09T09:00:28.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1266) [0x555f5c366f46] 2021-03-09T09:00:28.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x555f5c367766] 2021-03-09T09:00:28.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x555f5c395b0c] 2021-03-09T09:00:28.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 16: (DispatchQueue::entry()+0x126a) [0x7fc9ca7a30ea] 2021-03-09T09:00:28.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc9ca850ae1] 2021-03-09T09:00:28.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 18: (Thread::_entry_func(void*)+0xd) [0x7fc9ca65041d] 2021-03-09T09:00:28.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 19: /lib64/libpthread.so.0(+0x814a) [0x7fc9c804f14a] 2021-03-09T09:00:28.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 20: clone() 2021-03-09T09:00:28.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: debug 0> 2021-03-09T09:00:28.319+0000 7fc9bc9e6700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:00:28.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: in thread 7fc9bc9e6700 thread_name:ms_dispatch 2021-03-09T09:00:28.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:28.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc9c8059b20] 2021-03-09T09:00:28.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 2: gsignal() 2021-03-09T09:00:28.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 3: abort() 2021-03-09T09:00:28.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fc9ca569229] 2021-03-09T09:00:28.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fc9ca5693f2] 2021-03-09T09:00:28.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x555f5c48097c] 2021-03-09T09:00:28.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x555f5c4bfced] 2021-03-09T09:00:28.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x555f5c4e7b5a] 2021-03-09T09:00:28.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x555f5c4f3054] 2021-03-09T09:00:28.664 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x555f5c4f72c3] 2021-03-09T09:00:28.664 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x555f5c478c2d] 2021-03-09T09:00:28.664 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x555f5c361c74] 2021-03-09T09:00:28.664 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x555f5c3664d9] 2021-03-09T09:00:28.664 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x555f5c367766] 2021-03-09T09:00:28.665 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 15: (Monitor::handle_forward(boost::intrusive_ptr)+0x941) [0x555f5c368f91] 2021-03-09T09:00:28.665 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1266) [0x555f5c366f46] 2021-03-09T09:00:28.665 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 17: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x555f5c367766] 2021-03-09T09:00:28.665 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x555f5c395b0c] 2021-03-09T09:00:28.665 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 19: (DispatchQueue::entry()+0x126a) [0x7fc9ca7a30ea] 2021-03-09T09:00:28.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc9ca850ae1] 2021-03-09T09:00:28.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc9ca65041d] 2021-03-09T09:00:28.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 22: /lib64/libpthread.so.0(+0x814a) [0x7fc9c804f14a] 2021-03-09T09:00:28.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: 23: clone() 2021-03-09T09:00:28.667 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:28 smithi005 bash[25433]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:00:29.365 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:29 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Main process exited, code=exited, status=134/n/a 2021-03-09T09:00:29.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:29 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Failed with result 'exit-code'. 2021-03-09T09:00:29.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:00:29 smithi146 bash[13136]: debug 2021-03-09T09:00:29.354+0000 7fa560337700 -1 mon.b@2(electing) e3 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:00:34.365 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fc9257c5700 time 2021-03-09T09:00:34.115326+0000 2021-03-09T09:00:34.365 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:34.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:34.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fc934b4b1d8] 2021-03-09T09:00:34.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fc934b4b3f2] 2021-03-09T09:00:34.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55faf348597c] 2021-03-09T09:00:34.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55faf34c4ced] 2021-03-09T09:00:34.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55faf34ecb5a] 2021-03-09T09:00:34.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55faf34f8054] 2021-03-09T09:00:34.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55faf34fc2c3] 2021-03-09T09:00:34.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55faf347dc2d] 2021-03-09T09:00:34.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55faf33d20b4] 2021-03-09T09:00:34.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 10: (C_MonOp::finish(int)+0x49) [0x55faf3371069] 2021-03-09T09:00:34.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 11: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55faf3399da5] 2021-03-09T09:00:34.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 13: (Paxos::finish_round()+0x27b) [0x55faf347553b] 2021-03-09T09:00:34.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 14: (Paxos::commit_finish()+0x8c0) [0x55faf3477f50] 2021-03-09T09:00:34.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 15: (C_Committed::finish(int)+0x45) [0x55faf347bb85] 2021-03-09T09:00:34.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 16: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55faf347b8a8] 2021-03-09T09:00:34.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 18: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7fc934beaaa5] 2021-03-09T09:00:34.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 20: (Thread::_entry_func(void*)+0xd) [0x7fc934c3241d] 2021-03-09T09:00:34.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 21: /lib64/libpthread.so.0(+0x814a) [0x7fc93263114a] 2021-03-09T09:00:34.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 22: clone() 2021-03-09T09:00:34.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: debug 2021-03-09T09:00:34.119+0000 7fc9257c5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fc9257c5700 time 2021-03-09T09:00:34.115326+0000 2021-03-09T09:00:34.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:34.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:34.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fc934b4b1d8] 2021-03-09T09:00:34.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fc934b4b3f2] 2021-03-09T09:00:34.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55faf348597c] 2021-03-09T09:00:34.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55faf34c4ced] 2021-03-09T09:00:34.373 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55faf34ecb5a] 2021-03-09T09:00:34.373 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55faf34f8054] 2021-03-09T09:00:34.374 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55faf34fc2c3] 2021-03-09T09:00:34.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55faf347dc2d] 2021-03-09T09:00:34.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55faf33d20b4] 2021-03-09T09:00:34.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 10: (C_MonOp::finish(int)+0x49) [0x55faf3371069] 2021-03-09T09:00:34.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 11: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55faf3399da5] 2021-03-09T09:00:34.376 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 13: (Paxos::finish_round()+0x27b) [0x55faf347553b] 2021-03-09T09:00:34.376 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 14: (Paxos::commit_finish()+0x8c0) [0x55faf3477f50] 2021-03-09T09:00:34.376 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 15: (C_Committed::finish(int)+0x45) [0x55faf347bb85] 2021-03-09T09:00:34.376 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 16: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.377 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55faf347b8a8] 2021-03-09T09:00:34.377 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 18: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.377 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7fc934beaaa5] 2021-03-09T09:00:34.377 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 20: (Thread::_entry_func(void*)+0xd) [0x7fc934c3241d] 2021-03-09T09:00:34.377 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 21: /lib64/libpthread.so.0(+0x814a) [0x7fc93263114a] 2021-03-09T09:00:34.378 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 22: clone() 2021-03-09T09:00:34.378 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: *** Caught signal (Aborted) ** 2021-03-09T09:00:34.378 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: in thread 7fc9257c5700 thread_name:fn_monstore 2021-03-09T09:00:34.378 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:34.378 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc93263bb20] 2021-03-09T09:00:34.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 2: gsignal() 2021-03-09T09:00:34.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 3: abort() 2021-03-09T09:00:34.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fc934b4b229] 2021-03-09T09:00:34.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fc934b4b3f2] 2021-03-09T09:00:34.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55faf348597c] 2021-03-09T09:00:34.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55faf34c4ced] 2021-03-09T09:00:34.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55faf34ecb5a] 2021-03-09T09:00:34.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55faf34f8054] 2021-03-09T09:00:34.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55faf34fc2c3] 2021-03-09T09:00:34.381 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55faf347dc2d] 2021-03-09T09:00:34.381 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55faf33d20b4] 2021-03-09T09:00:34.381 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 13: (C_MonOp::finish(int)+0x49) [0x55faf3371069] 2021-03-09T09:00:34.381 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 14: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.381 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55faf3399da5] 2021-03-09T09:00:34.382 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 16: (Paxos::finish_round()+0x27b) [0x55faf347553b] 2021-03-09T09:00:34.382 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 17: (Paxos::commit_finish()+0x8c0) [0x55faf3477f50] 2021-03-09T09:00:34.382 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 18: (C_Committed::finish(int)+0x45) [0x55faf347bb85] 2021-03-09T09:00:34.382 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 19: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.382 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55faf347b8a8] 2021-03-09T09:00:34.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 21: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7fc934beaaa5] 2021-03-09T09:00:34.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 23: (Thread::_entry_func(void*)+0xd) [0x7fc934c3241d] 2021-03-09T09:00:34.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 24: /lib64/libpthread.so.0(+0x814a) [0x7fc93263114a] 2021-03-09T09:00:34.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 25: clone() 2021-03-09T09:00:34.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: debug 2021-03-09T09:00:34.123+0000 7fc9257c5700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:00:34.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: in thread 7fc9257c5700 thread_name:fn_monstore 2021-03-09T09:00:34.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:34.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc93263bb20] 2021-03-09T09:00:34.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 2: gsignal() 2021-03-09T09:00:34.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 3: abort() 2021-03-09T09:00:34.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fc934b4b229] 2021-03-09T09:00:34.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fc934b4b3f2] 2021-03-09T09:00:34.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55faf348597c] 2021-03-09T09:00:34.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55faf34c4ced] 2021-03-09T09:00:34.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55faf34ecb5a] 2021-03-09T09:00:34.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55faf34f8054] 2021-03-09T09:00:34.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55faf34fc2c3] 2021-03-09T09:00:34.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55faf347dc2d] 2021-03-09T09:00:34.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55faf33d20b4] 2021-03-09T09:00:34.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 13: (C_MonOp::finish(int)+0x49) [0x55faf3371069] 2021-03-09T09:00:34.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 14: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55faf3399da5] 2021-03-09T09:00:34.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 16: (Paxos::finish_round()+0x27b) [0x55faf347553b] 2021-03-09T09:00:34.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 17: (Paxos::commit_finish()+0x8c0) [0x55faf3477f50] 2021-03-09T09:00:34.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 18: (C_Committed::finish(int)+0x45) [0x55faf347bb85] 2021-03-09T09:00:34.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 19: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55faf347b8a8] 2021-03-09T09:00:34.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 21: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.389 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7fc934beaaa5] 2021-03-09T09:00:34.389 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 23: (Thread::_entry_func(void*)+0xd) [0x7fc934c3241d] 2021-03-09T09:00:34.389 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 24: /lib64/libpthread.so.0(+0x814a) [0x7fc93263114a] 2021-03-09T09:00:34.389 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 25: clone() 2021-03-09T09:00:34.389 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:00:34.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: debug -1> 2021-03-09T09:00:34.119+0000 7fc9257c5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fc9257c5700 time 2021-03-09T09:00:34.115326+0000 2021-03-09T09:00:34.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:34.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:34.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fc934b4b1d8] 2021-03-09T09:00:34.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fc934b4b3f2] 2021-03-09T09:00:34.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55faf348597c] 2021-03-09T09:00:34.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55faf34c4ced] 2021-03-09T09:00:34.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55faf34ecb5a] 2021-03-09T09:00:34.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55faf34f8054] 2021-03-09T09:00:34.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55faf34fc2c3] 2021-03-09T09:00:34.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55faf347dc2d] 2021-03-09T09:00:34.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55faf33d20b4] 2021-03-09T09:00:34.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 10: (C_MonOp::finish(int)+0x49) [0x55faf3371069] 2021-03-09T09:00:34.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 11: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55faf3399da5] 2021-03-09T09:00:34.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 13: (Paxos::finish_round()+0x27b) [0x55faf347553b] 2021-03-09T09:00:34.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 14: (Paxos::commit_finish()+0x8c0) [0x55faf3477f50] 2021-03-09T09:00:34.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 15: (C_Committed::finish(int)+0x45) [0x55faf347bb85] 2021-03-09T09:00:34.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 16: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55faf347b8a8] 2021-03-09T09:00:34.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 18: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7fc934beaaa5] 2021-03-09T09:00:34.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 20: (Thread::_entry_func(void*)+0xd) [0x7fc934c3241d] 2021-03-09T09:00:34.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 21: /lib64/libpthread.so.0(+0x814a) [0x7fc93263114a] 2021-03-09T09:00:34.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 22: clone() 2021-03-09T09:00:34.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: debug 0> 2021-03-09T09:00:34.123+0000 7fc9257c5700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:00:34.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: in thread 7fc9257c5700 thread_name:fn_monstore 2021-03-09T09:00:34.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:34.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc93263bb20] 2021-03-09T09:00:34.396 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 2: gsignal() 2021-03-09T09:00:34.396 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 3: abort() 2021-03-09T09:00:34.396 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fc934b4b229] 2021-03-09T09:00:34.396 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fc934b4b3f2] 2021-03-09T09:00:34.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55faf348597c] 2021-03-09T09:00:34.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55faf34c4ced] 2021-03-09T09:00:34.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55faf34ecb5a] 2021-03-09T09:00:34.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55faf34f8054] 2021-03-09T09:00:34.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55faf34fc2c3] 2021-03-09T09:00:34.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55faf347dc2d] 2021-03-09T09:00:34.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55faf33d20b4] 2021-03-09T09:00:34.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 13: (C_MonOp::finish(int)+0x49) [0x55faf3371069] 2021-03-09T09:00:34.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 14: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55faf3399da5] 2021-03-09T09:00:34.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 16: (Paxos::finish_round()+0x27b) [0x55faf347553b] 2021-03-09T09:00:34.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 17: (Paxos::commit_finish()+0x8c0) [0x55faf3477f50] 2021-03-09T09:00:34.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 18: (C_Committed::finish(int)+0x45) [0x55faf347bb85] 2021-03-09T09:00:34.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 19: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55faf347b8a8] 2021-03-09T09:00:34.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 21: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7fc934beaaa5] 2021-03-09T09:00:34.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 23: (Thread::_entry_func(void*)+0xd) [0x7fc934c3241d] 2021-03-09T09:00:34.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 24: /lib64/libpthread.so.0(+0x814a) [0x7fc93263114a] 2021-03-09T09:00:34.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 25: clone() 2021-03-09T09:00:34.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:00:34.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: debug -1> 2021-03-09T09:00:34.119+0000 7fc9257c5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fc9257c5700 time 2021-03-09T09:00:34.115326+0000 2021-03-09T09:00:34.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:34.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:34.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fc934b4b1d8] 2021-03-09T09:00:34.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fc934b4b3f2] 2021-03-09T09:00:34.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55faf348597c] 2021-03-09T09:00:34.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55faf34c4ced] 2021-03-09T09:00:34.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55faf34ecb5a] 2021-03-09T09:00:34.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55faf34f8054] 2021-03-09T09:00:34.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55faf34fc2c3] 2021-03-09T09:00:34.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55faf347dc2d] 2021-03-09T09:00:34.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55faf33d20b4] 2021-03-09T09:00:34.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 10: (C_MonOp::finish(int)+0x49) [0x55faf3371069] 2021-03-09T09:00:34.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 11: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55faf3399da5] 2021-03-09T09:00:34.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 13: (Paxos::finish_round()+0x27b) [0x55faf347553b] 2021-03-09T09:00:34.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 14: (Paxos::commit_finish()+0x8c0) [0x55faf3477f50] 2021-03-09T09:00:34.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 15: (C_Committed::finish(int)+0x45) [0x55faf347bb85] 2021-03-09T09:00:34.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 16: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55faf347b8a8] 2021-03-09T09:00:34.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 18: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7fc934beaaa5] 2021-03-09T09:00:34.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 20: (Thread::_entry_func(void*)+0xd) [0x7fc934c3241d] 2021-03-09T09:00:34.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 21: /lib64/libpthread.so.0(+0x814a) [0x7fc93263114a] 2021-03-09T09:00:34.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 22: clone() 2021-03-09T09:00:34.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: debug 0> 2021-03-09T09:00:34.123+0000 7fc9257c5700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:00:34.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: in thread 7fc9257c5700 thread_name:fn_monstore 2021-03-09T09:00:34.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:34.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc93263bb20] 2021-03-09T09:00:34.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 2: gsignal() 2021-03-09T09:00:34.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 3: abort() 2021-03-09T09:00:34.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fc934b4b229] 2021-03-09T09:00:34.408 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fc934b4b3f2] 2021-03-09T09:00:34.408 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55faf348597c] 2021-03-09T09:00:34.408 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55faf34c4ced] 2021-03-09T09:00:34.408 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55faf34ecb5a] 2021-03-09T09:00:34.408 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55faf34f8054] 2021-03-09T09:00:34.409 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55faf34fc2c3] 2021-03-09T09:00:34.409 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55faf347dc2d] 2021-03-09T09:00:34.409 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55faf33d20b4] 2021-03-09T09:00:34.409 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 13: (C_MonOp::finish(int)+0x49) [0x55faf3371069] 2021-03-09T09:00:34.409 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 14: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.410 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55faf3399da5] 2021-03-09T09:00:34.410 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 16: (Paxos::finish_round()+0x27b) [0x55faf347553b] 2021-03-09T09:00:34.410 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 17: (Paxos::commit_finish()+0x8c0) [0x55faf3477f50] 2021-03-09T09:00:34.410 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 18: (C_Committed::finish(int)+0x45) [0x55faf347bb85] 2021-03-09T09:00:34.410 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 19: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55faf347b8a8] 2021-03-09T09:00:34.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 21: (Context::complete(int)+0xd) [0x55faf336e49d] 2021-03-09T09:00:34.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7fc934beaaa5] 2021-03-09T09:00:34.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 23: (Thread::_entry_func(void*)+0xd) [0x7fc934c3241d] 2021-03-09T09:00:34.412 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 24: /lib64/libpthread.so.0(+0x814a) [0x7fc93263114a] 2021-03-09T09:00:34.412 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: 25: clone() 2021-03-09T09:00:34.412 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 bash[25305]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:00:34.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:00:34 smithi146 bash[13136]: debug 2021-03-09T09:00:34.358+0000 7fa560337700 -1 mon.b@2(peon) e3 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:00:35.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Main process exited, code=exited, status=134/n/a 2021-03-09T09:00:35.365 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:34 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Failed with result 'exit-code'. 2021-03-09T09:00:39.614 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:39 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Scheduled restart job, restart counter is at 2. 2021-03-09T09:00:39.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:39 smithi005 systemd[1]: Stopped Ceph mon.c for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:00:39.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:39 smithi005 systemd[1]: Started Ceph mon.c for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:00:39.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:00:39 smithi146 bash[13136]: debug 2021-03-09T09:00:39.354+0000 7fa560337700 -1 mon.b@2(peon) e3 get_health_metrics reporting 17 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:00:40.291 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.279+0000 7f00dfea6700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-09T09:00:40.292 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.279+0000 7f00dfea6700 0 ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev), process ceph-mon, pid 6 2021-03-09T09:00:40.292 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.279+0000 7f00dfea6700 0 pidfile_write: ignore empty --pid-file 2021-03-09T09:00:40.292 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 0 load: jerasure load: lrc load: isa 2021-03-09T09:00:40.292 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: RocksDB version: 6.8.1 2021-03-09T09:00:40.293 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-09T09:00:40.293 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Compile date Mar 9 2021 2021-03-09T09:00:40.293 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: DB SUMMARY 2021-03-09T09:00:40.293 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: CURRENT file: CURRENT 2021-03-09T09:00:40.294 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-09T09:00:40.294 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: MANIFEST file: MANIFEST-000008 size: 203 Bytes 2021-03-09T09:00:40.294 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 2, files: 000004.sst 000007.sst 2021-03-09T09:00:40.294 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000009.log size: 11687 ; 2021-03-09T09:00:40.295 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.error_if_exists: 0 2021-03-09T09:00:40.295 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.create_if_missing: 0 2021-03-09T09:00:40.295 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.paranoid_checks: 1 2021-03-09T09:00:40.295 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.env: 0x557b416d31c0 2021-03-09T09:00:40.296 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.fs: Posix File System 2021-03-09T09:00:40.296 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.info_log: 0x557b4b2d7120 2021-03-09T09:00:40.296 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-09T09:00:40.296 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.statistics: (nil) 2021-03-09T09:00:40.296 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.use_fsync: 0 2021-03-09T09:00:40.297 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_log_file_size: 0 2021-03-09T09:00:40.297 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-09T09:00:40.297 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-09T09:00:40.297 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-09T09:00:40.298 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-09T09:00:40.299 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.allow_fallocate: 1 2021-03-09T09:00:40.300 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-09T09:00:40.300 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-09T09:00:40.300 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.use_direct_reads: 0 2021-03-09T09:00:40.300 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-09T09:00:40.301 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-09T09:00:40.301 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.db_log_dir: 2021-03-09T09:00:40.301 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2021-03-09T09:00:40.302 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-09T09:00:40.302 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_subcompactions: 1 2021-03-09T09:00:40.302 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_background_flushes: -1 2021-03-09T09:00:40.302 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-09T09:00:40.303 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-09T09:00:40.303 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-09T09:00:40.303 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-09T09:00:40.303 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-09T09:00:40.304 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-09T09:00:40.304 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-09T09:00:40.304 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.write_buffer_manager: 0x557b4b32bce0 2021-03-09T09:00:40.305 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-09T09:00:40.305 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-09T09:00:40.306 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-09T09:00:40.306 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-09T09:00:40.307 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-09T09:00:40.307 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-09T09:00:40.308 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-09T09:00:40.308 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-09T09:00:40.309 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-09T09:00:40.309 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.unordered_write: 0 2021-03-09T09:00:40.310 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-09T09:00:40.310 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-09T09:00:40.310 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-09T09:00:40.311 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-09T09:00:40.311 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.row_cache: None 2021-03-09T09:00:40.312 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.wal_filter: None 2021-03-09T09:00:40.312 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-09T09:00:40.313 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-09T09:00:40.313 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.preserve_deletes: 0 2021-03-09T09:00:40.314 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.two_write_queues: 0 2021-03-09T09:00:40.314 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-09T09:00:40.315 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.atomic_flush: 0 2021-03-09T09:00:40.315 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-09T09:00:40.316 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-09T09:00:40.316 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-09T09:00:40.316 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.log_readahead_size: 0 2021-03-09T09:00:40.317 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-09T09:00:40.317 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_background_jobs: 2 2021-03-09T09:00:40.318 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_background_compactions: -1 2021-03-09T09:00:40.318 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-09T09:00:40.319 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-09T09:00:40.319 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-09T09:00:40.320 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-09T09:00:40.320 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-09T09:00:40.320 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-09T09:00:40.321 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-09T09:00:40.321 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-09T09:00:40.322 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_open_files: -1 2021-03-09T09:00:40.322 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-09T09:00:40.323 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-09T09:00:40.323 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-09T09:00:40.324 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-09T09:00:40.324 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Compression algorithms supported: 2021-03-09T09:00:40.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-09T09:00:40.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: kZSTD supported: 0 2021-03-09T09:00:40.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: kXpressCompression supported: 0 2021-03-09T09:00:40.326 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-09T09:00:40.326 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: kLZ4Compression supported: 1 2021-03-09T09:00:40.327 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: kBZip2Compression supported: 0 2021-03-09T09:00:40.327 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: kZlibCompression supported: 1 2021-03-09T09:00:40.327 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: kSnappyCompression supported: 1 2021-03-09T09:00:40.328 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-09T09:00:40.328 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000008 2021-03-09T09:00:40.328 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-09T09:00:40.329 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-09T09:00:40.329 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.merge_operator: 2021-03-09T09:00:40.329 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compaction_filter: None 2021-03-09T09:00:40.329 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-09T09:00:40.330 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-09T09:00:40.330 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-09T09:00:40.330 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557b4307a230) 2021-03-09T09:00:40.331 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: cache_index_and_filter_blocks: 1 2021-03-09T09:00:40.331 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-09T09:00:40.332 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-09T09:00:40.332 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: pin_top_level_index_and_filter: 1 2021-03-09T09:00:40.333 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: index_type: 0 2021-03-09T09:00:40.333 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: data_block_index_type: 0 2021-03-09T09:00:40.333 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: index_shortening: 1 2021-03-09T09:00:40.334 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: data_block_hash_table_util_ratio: 0.750000 2021-03-09T09:00:40.334 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: hash_index_allow_collision: 1 2021-03-09T09:00:40.335 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: checksum: 1 2021-03-09T09:00:40.335 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: no_block_cache: 0 2021-03-09T09:00:40.336 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: block_cache: 0x557b4b2b2510 2021-03-09T09:00:40.336 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: block_cache_name: BinnedLRUCache 2021-03-09T09:00:40.336 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: block_cache_options: 2021-03-09T09:00:40.337 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: capacity : 536870912 2021-03-09T09:00:40.337 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: num_shard_bits : 4 2021-03-09T09:00:40.338 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: strict_capacity_limit : 0 2021-03-09T09:00:40.338 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: high_pri_pool_ratio: 0.000 2021-03-09T09:00:40.338 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: block_cache_compressed: (nil) 2021-03-09T09:00:40.339 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: persistent_cache: (nil) 2021-03-09T09:00:40.339 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: block_size: 4096 2021-03-09T09:00:40.340 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: block_size_deviation: 10 2021-03-09T09:00:40.340 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: block_restart_interval: 16 2021-03-09T09:00:40.341 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: index_block_restart_interval: 1 2021-03-09T09:00:40.341 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: metadata_block_size: 4096 2021-03-09T09:00:40.342 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: partition_filters: 0 2021-03-09T09:00:40.342 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: use_delta_encoding: 1 2021-03-09T09:00:40.342 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-09T09:00:40.343 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: whole_key_filtering: 1 2021-03-09T09:00:40.343 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: verify_compression: 0 2021-03-09T09:00:40.344 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: read_amp_bytes_per_bit: 0 2021-03-09T09:00:40.344 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: format_version: 2 2021-03-09T09:00:40.344 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: enable_index_compression: 1 2021-03-09T09:00:40.345 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: block_align: 0 2021-03-09T09:00:40.345 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-09T09:00:40.346 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-09T09:00:40.346 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compression: NoCompression 2021-03-09T09:00:40.347 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-09T09:00:40.347 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-09T09:00:40.347 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-09T09:00:40.347 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.num_levels: 7 2021-03-09T09:00:40.347 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-09T09:00:40.348 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-09T09:00:40.348 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-09T09:00:40.348 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-09T09:00:40.348 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-09T09:00:40.348 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-09T09:00:40.349 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-09T09:00:40.349 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:00:40.349 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-09T09:00:40.349 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-09T09:00:40.350 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-09T09:00:40.350 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-09T09:00:40.350 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-09T09:00:40.350 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:00:40.350 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-09T09:00:40.351 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-09T09:00:40.351 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-09T09:00:40.351 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-09T09:00:40.351 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-09T09:00:40.351 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-09T09:00:40.352 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-09T09:00:40.352 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-09T09:00:40.352 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-09T09:00:40.352 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-09T09:00:40.352 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-09T09:00:40.353 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-09T09:00:40.353 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-09T09:00:40.353 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-09T09:00:40.353 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-09T09:00:40.353 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-09T09:00:40.354 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-09T09:00:40.354 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-09T09:00:40.354 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-09T09:00:40.354 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-09T09:00:40.355 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-09T09:00:40.355 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-09T09:00:40.355 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-09T09:00:40.355 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-09T09:00:40.355 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-09T09:00:40.356 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-09T09:00:40.356 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-09T09:00:40.356 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-09T09:00:40.356 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-09T09:00:40.356 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-09T09:00:40.357 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-09T09:00:40.357 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-09T09:00:40.357 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-09T09:00:40.357 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.table_properties_collectors: 2021-03-09T09:00:40.357 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.inplace_update_support: 0 2021-03-09T09:00:40.358 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-09T09:00:40.358 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-09T09:00:40.358 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-09T09:00:40.358 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-09T09:00:40.358 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.bloom_locality: 0 2021-03-09T09:00:40.359 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.max_successive_merges: 0 2021-03-09T09:00:40.359 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-09T09:00:40.359 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-09T09:00:40.359 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-09T09:00:40.359 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-09T09:00:40.360 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.ttl: 2592000 2021-03-09T09:00:40.360 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.283+0000 7f00dfea6700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-09T09:00:40.360 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.287+0000 7f00dfea6700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 10, last_sequence is 2919, log_number is 7,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-09T09:00:40.360 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.287+0000 7f00dfea6700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 7 2021-03-09T09:00:40.361 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.287+0000 7f00dfea6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280440289111, "job": 1, "event": "recovery_started", "log_files": [9]} 2021-03-09T09:00:40.361 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.287+0000 7f00dfea6700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #9 mode 2 2021-03-09T09:00:40.361 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.287+0000 7f00dfea6700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-09T09:00:40.361 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.287+0000 7f00dfea6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280440289981, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 10, "file_size": 11504, "table_properties": {"data_size": 10542, "index_size": 87, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 342, "raw_average_key_size": 24, "raw_value_size": 10196, "raw_average_value_size": 728, "num_data_blocks": 3, "num_entries": 14, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1615280440, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-09T09:00:40.361 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.287+0000 7f00dfea6700 4 rocksdb: [version_set.cc:3826] Creating manifest 11 2021-03-09T09:00:40.362 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.287+0000 7f00dfea6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280440290608, "job": 1, "event": "recovery_finished"} 2021-03-09T09:00:40.362 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.287+0000 7f00dfea6700 4 rocksdb: DB pointer 0x557b4b377800 2021-03-09T09:00:40.362 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.291+0000 7f00ceeb6700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-09T09:00:40.362 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.291+0000 7f00ceeb6700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-09T09:00:40.362 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: ** DB Stats ** 2021-03-09T09:00:40.365 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:00:40.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-09T09:00:40.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-09T09:00:40.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:00:40.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-09T09:00:40.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-09T09:00:40.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:00:40.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: ** Compaction Stats [default] ** 2021-03-09T09:00:40.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:40.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:40.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: L0 3/0 7.67 MB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 17.8 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:40.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Sum 3/0 7.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 17.8 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:40.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 17.8 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:40.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: ** Compaction Stats [default] ** 2021-03-09T09:00:40.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:40.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:40.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 17.8 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:40.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:00:40.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T09:00:40.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:00:40.374 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:00:40.374 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:00:40.375 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:00:40.375 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Cumulative compaction: 0.00 GB write, 2.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:40.376 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Interval compaction: 0.00 GB write, 2.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:40.376 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:00:40.376 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:00:40.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: ** Compaction Stats [default] ** 2021-03-09T09:00:40.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:40.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:40.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: L0 3/0 7.67 MB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 17.8 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:40.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Sum 3/0 7.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 17.8 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:40.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:00:40.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: ** Compaction Stats [default] ** 2021-03-09T09:00:40.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:40.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:40.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 17.8 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:40.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:00:40.382 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T09:00:40.382 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:00:40.382 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:00:40.383 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:00:40.383 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:00:40.384 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Cumulative compaction: 0.00 GB write, 2.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:40.384 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:40.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:00:40.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:00:40.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.291+0000 7f00dfea6700 0 starting mon.c rank 1 at public addrs [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] at bind addrs [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:00:40.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.295+0000 7f00dfea6700 1 mon.c@-1(???) e3 preinit fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:00:40.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.295+0000 7f00dfea6700 0 mon.c@-1(???).mds e1 new map 2021-03-09T09:00:40.387 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.295+0000 7f00dfea6700 0 mon.c@-1(???).mds e1 print_map 2021-03-09T09:00:40.387 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: e1 2021-03-09T09:00:40.388 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: enable_multiple, ever_enabled_multiple: 1,1 2021-03-09T09:00:40.389 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-03-09T09:00:40.389 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: legacy client fscid: -1 2021-03-09T09:00:40.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: 2021-03-09T09:00:40.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: No filesystems configured 2021-03-09T09:00:40.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.295+0000 7f00dfea6700 0 mon.c@-1(???).osd e18 crush map has features 3314932999778484224, adjusting msgr requires 2021-03-09T09:00:40.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.295+0000 7f00dfea6700 0 mon.c@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:00:40.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.295+0000 7f00dfea6700 0 mon.c@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:00:40.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.295+0000 7f00dfea6700 0 mon.c@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:00:40.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: audit 2021-03-09T08:59:48.920210+0000 mon.a (mon.0) 269 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T09:00:40.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: cluster 2021-03-09T08:59:48.920524+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-09T09:00:40.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: audit 2021-03-09T08:59:48.921424+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:00:40.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: audit 2021-03-09T08:59:48.930506+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:00:40.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.299+0000 7f00dfea6700 1 mon.c@-1(???).paxosservice(auth 1..10) refresh upgraded, format 0 -> 3 2021-03-09T09:00:40.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.299+0000 7f00dfea6700 20 expand_channel_meta expand map: {default=false} 2021-03-09T09:00:40.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.299+0000 7f00dfea6700 20 expand_channel_meta from 'false' to 'false' 2021-03-09T09:00:40.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.299+0000 7f00dfea6700 20 expand_channel_meta expanded map: {default=false} 2021-03-09T09:00:40.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.299+0000 7f00dfea6700 20 expand_channel_meta expand map: {default=info} 2021-03-09T09:00:40.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.299+0000 7f00dfea6700 20 expand_channel_meta from 'info' to 'info' 2021-03-09T09:00:40.394 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.299+0000 7f00dfea6700 20 expand_channel_meta expanded map: {default=info} 2021-03-09T09:00:40.394 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.299+0000 7f00dfea6700 20 expand_channel_meta expand map: {default=daemon} 2021-03-09T09:00:40.394 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.299+0000 7f00dfea6700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-03-09T09:00:40.394 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.299+0000 7f00dfea6700 20 expand_channel_meta expanded map: {default=daemon} 2021-03-09T09:00:40.394 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.299+0000 7f00dfea6700 20 expand_channel_meta expand map: {default=debug} 2021-03-09T09:00:40.395 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.299+0000 7f00dfea6700 20 expand_channel_meta from 'debug' to 'debug' 2021-03-09T09:00:40.395 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:40 smithi005 bash[25598]: debug 2021-03-09T09:00:40.299+0000 7f00dfea6700 20 expand_channel_meta expanded map: {default=debug} 2021-03-09T09:00:44.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:00:44 smithi146 bash[13136]: debug 2021-03-09T09:00:44.354+0000 7fa560337700 -1 mon.b@2(electing) e3 get_health_metrics reporting 28 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:00:45.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:45 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Scheduled restart job, restart counter is at 2. 2021-03-09T09:00:45.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:45 smithi005 systemd[1]: Stopped Ceph mon.a for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:00:45.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:45 smithi005 systemd[1]: Started Ceph mon.a for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:00:46.287 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f00cfeb8700 time 2021-03-09T09:00:46.007450+0000 2021-03-09T09:00:46.288 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:46.288 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:46.288 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f00df23e1d8] 2021-03-09T09:00:46.288 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f00df23e3f2] 2021-03-09T09:00:46.289 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x557b40f3597c] 2021-03-09T09:00:46.289 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x557b40f74ced] 2021-03-09T09:00:46.289 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x557b40f9cb5a] 2021-03-09T09:00:46.289 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x557b40fa8054] 2021-03-09T09:00:46.290 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x557b40fac2c3] 2021-03-09T09:00:46.290 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x557b40f2dc2d] 2021-03-09T09:00:46.290 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x557b40e820b4] 2021-03-09T09:00:46.291 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 10: (C_MonOp::finish(int)+0x49) [0x557b40e21069] 2021-03-09T09:00:46.291 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 11: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.291 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x557b40e49da5] 2021-03-09T09:00:46.291 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 13: (Paxos::finish_round()+0x27b) [0x557b40f2553b] 2021-03-09T09:00:46.291 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 14: (Paxos::commit_finish()+0x8c0) [0x557b40f27f50] 2021-03-09T09:00:46.292 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 15: (C_Committed::finish(int)+0x45) [0x557b40f2bb85] 2021-03-09T09:00:46.292 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 16: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.292 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x557b40f2b8a8] 2021-03-09T09:00:46.292 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 18: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.293 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f00df2ddaa5] 2021-03-09T09:00:46.293 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 20: (Thread::_entry_func(void*)+0xd) [0x7f00df32541d] 2021-03-09T09:00:46.293 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f00dcd2414a] 2021-03-09T09:00:46.293 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 22: clone() 2021-03-09T09:00:46.294 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: *** Caught signal (Aborted) ** 2021-03-09T09:00:46.294 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: in thread 7f00cfeb8700 thread_name:fn_monstore 2021-03-09T09:00:46.294 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: debug 2021-03-09T09:00:46.007+0000 7f00cfeb8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f00cfeb8700 time 2021-03-09T09:00:46.007450+0000 2021-03-09T09:00:46.294 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:46.295 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:46.295 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f00df23e1d8] 2021-03-09T09:00:46.295 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f00df23e3f2] 2021-03-09T09:00:46.295 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x557b40f3597c] 2021-03-09T09:00:46.296 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x557b40f74ced] 2021-03-09T09:00:46.296 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x557b40f9cb5a] 2021-03-09T09:00:46.298 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x557b40fa8054] 2021-03-09T09:00:46.298 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x557b40fac2c3] 2021-03-09T09:00:46.298 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x557b40f2dc2d] 2021-03-09T09:00:46.299 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x557b40e820b4] 2021-03-09T09:00:46.299 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 10: (C_MonOp::finish(int)+0x49) [0x557b40e21069] 2021-03-09T09:00:46.299 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 11: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.300 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x557b40e49da5] 2021-03-09T09:00:46.300 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 13: (Paxos::finish_round()+0x27b) [0x557b40f2553b] 2021-03-09T09:00:46.300 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 14: (Paxos::commit_finish()+0x8c0) [0x557b40f27f50] 2021-03-09T09:00:46.300 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 15: (C_Committed::finish(int)+0x45) [0x557b40f2bb85] 2021-03-09T09:00:46.300 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 16: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.301 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x557b40f2b8a8] 2021-03-09T09:00:46.301 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 18: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.301 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f00df2ddaa5] 2021-03-09T09:00:46.302 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 20: (Thread::_entry_func(void*)+0xd) [0x7f00df32541d] 2021-03-09T09:00:46.302 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f00dcd2414a] 2021-03-09T09:00:46.302 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 22: clone() 2021-03-09T09:00:46.302 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:46.303 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f00dcd2eb20] 2021-03-09T09:00:46.303 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 2: gsignal() 2021-03-09T09:00:46.303 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 3: abort() 2021-03-09T09:00:46.303 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f00df23e229] 2021-03-09T09:00:46.304 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f00df23e3f2] 2021-03-09T09:00:46.304 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x557b40f3597c] 2021-03-09T09:00:46.304 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x557b40f74ced] 2021-03-09T09:00:46.305 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x557b40f9cb5a] 2021-03-09T09:00:46.305 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x557b40fa8054] 2021-03-09T09:00:46.305 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x557b40fac2c3] 2021-03-09T09:00:46.306 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x557b40f2dc2d] 2021-03-09T09:00:46.306 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x557b40e820b4] 2021-03-09T09:00:46.306 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 13: (C_MonOp::finish(int)+0x49) [0x557b40e21069] 2021-03-09T09:00:46.306 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 14: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.306 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x557b40e49da5] 2021-03-09T09:00:46.307 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 16: (Paxos::finish_round()+0x27b) [0x557b40f2553b] 2021-03-09T09:00:46.307 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 17: (Paxos::commit_finish()+0x8c0) [0x557b40f27f50] 2021-03-09T09:00:46.307 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 18: (C_Committed::finish(int)+0x45) [0x557b40f2bb85] 2021-03-09T09:00:46.307 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 19: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.308 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x557b40f2b8a8] 2021-03-09T09:00:46.308 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 21: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.308 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f00df2ddaa5] 2021-03-09T09:00:46.308 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 23: (Thread::_entry_func(void*)+0xd) [0x7f00df32541d] 2021-03-09T09:00:46.309 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f00dcd2414a] 2021-03-09T09:00:46.309 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 25: clone() 2021-03-09T09:00:46.309 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: debug 2021-03-09T09:00:46.011+0000 7f00cfeb8700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:00:46.309 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: in thread 7f00cfeb8700 thread_name:fn_monstore 2021-03-09T09:00:46.310 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:46.310 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f00dcd2eb20] 2021-03-09T09:00:46.310 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 2: gsignal() 2021-03-09T09:00:46.310 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 3: abort() 2021-03-09T09:00:46.310 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f00df23e229] 2021-03-09T09:00:46.311 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f00df23e3f2] 2021-03-09T09:00:46.311 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x557b40f3597c] 2021-03-09T09:00:46.311 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x557b40f74ced] 2021-03-09T09:00:46.311 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x557b40f9cb5a] 2021-03-09T09:00:46.312 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x557b40fa8054] 2021-03-09T09:00:46.312 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x557b40fac2c3] 2021-03-09T09:00:46.312 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x557b40f2dc2d] 2021-03-09T09:00:46.312 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x557b40e820b4] 2021-03-09T09:00:46.313 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 13: (C_MonOp::finish(int)+0x49) [0x557b40e21069] 2021-03-09T09:00:46.313 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 14: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.313 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x557b40e49da5] 2021-03-09T09:00:46.313 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 16: (Paxos::finish_round()+0x27b) [0x557b40f2553b] 2021-03-09T09:00:46.313 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 17: (Paxos::commit_finish()+0x8c0) [0x557b40f27f50] 2021-03-09T09:00:46.314 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 18: (C_Committed::finish(int)+0x45) [0x557b40f2bb85] 2021-03-09T09:00:46.314 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 19: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.314 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x557b40f2b8a8] 2021-03-09T09:00:46.314 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 21: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.315 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f00df2ddaa5] 2021-03-09T09:00:46.315 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 23: (Thread::_entry_func(void*)+0xd) [0x7f00df32541d] 2021-03-09T09:00:46.315 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f00dcd2414a] 2021-03-09T09:00:46.315 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 25: clone() 2021-03-09T09:00:46.315 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:00:46.316 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: debug -1> 2021-03-09T09:00:46.007+0000 7f00cfeb8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f00cfeb8700 time 2021-03-09T09:00:46.007450+0000 2021-03-09T09:00:46.316 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:46.316 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:46.316 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f00df23e1d8] 2021-03-09T09:00:46.316 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f00df23e3f2] 2021-03-09T09:00:46.317 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x557b40f3597c] 2021-03-09T09:00:46.317 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x557b40f74ced] 2021-03-09T09:00:46.317 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x557b40f9cb5a] 2021-03-09T09:00:46.317 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x557b40fa8054] 2021-03-09T09:00:46.318 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x557b40fac2c3] 2021-03-09T09:00:46.318 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x557b40f2dc2d] 2021-03-09T09:00:46.318 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x557b40e820b4] 2021-03-09T09:00:46.318 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 10: (C_MonOp::finish(int)+0x49) [0x557b40e21069] 2021-03-09T09:00:46.318 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 11: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.319 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x557b40e49da5] 2021-03-09T09:00:46.319 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 13: (Paxos::finish_round()+0x27b) [0x557b40f2553b] 2021-03-09T09:00:46.319 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 14: (Paxos::commit_finish()+0x8c0) [0x557b40f27f50] 2021-03-09T09:00:46.319 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 15: (C_Committed::finish(int)+0x45) [0x557b40f2bb85] 2021-03-09T09:00:46.320 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 16: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.320 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x557b40f2b8a8] 2021-03-09T09:00:46.320 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 18: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.320 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f00df2ddaa5] 2021-03-09T09:00:46.320 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 20: (Thread::_entry_func(void*)+0xd) [0x7f00df32541d] 2021-03-09T09:00:46.321 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f00dcd2414a] 2021-03-09T09:00:46.321 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 22: clone() 2021-03-09T09:00:46.321 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: debug 0> 2021-03-09T09:00:46.011+0000 7f00cfeb8700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:00:46.321 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: in thread 7f00cfeb8700 thread_name:fn_monstore 2021-03-09T09:00:46.321 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:46.322 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f00dcd2eb20] 2021-03-09T09:00:46.322 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 2: gsignal() 2021-03-09T09:00:46.322 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 3: abort() 2021-03-09T09:00:46.322 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f00df23e229] 2021-03-09T09:00:46.323 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f00df23e3f2] 2021-03-09T09:00:46.323 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x557b40f3597c] 2021-03-09T09:00:46.323 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x557b40f74ced] 2021-03-09T09:00:46.323 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x557b40f9cb5a] 2021-03-09T09:00:46.323 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x557b40fa8054] 2021-03-09T09:00:46.324 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x557b40fac2c3] 2021-03-09T09:00:46.324 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x557b40f2dc2d] 2021-03-09T09:00:46.324 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x557b40e820b4] 2021-03-09T09:00:46.324 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 13: (C_MonOp::finish(int)+0x49) [0x557b40e21069] 2021-03-09T09:00:46.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 14: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x557b40e49da5] 2021-03-09T09:00:46.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 16: (Paxos::finish_round()+0x27b) [0x557b40f2553b] 2021-03-09T09:00:46.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 17: (Paxos::commit_finish()+0x8c0) [0x557b40f27f50] 2021-03-09T09:00:46.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 18: (C_Committed::finish(int)+0x45) [0x557b40f2bb85] 2021-03-09T09:00:46.326 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 19: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.326 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x557b40f2b8a8] 2021-03-09T09:00:46.326 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 21: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.326 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f00df2ddaa5] 2021-03-09T09:00:46.327 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 23: (Thread::_entry_func(void*)+0xd) [0x7f00df32541d] 2021-03-09T09:00:46.327 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f00dcd2414a] 2021-03-09T09:00:46.327 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 25: clone() 2021-03-09T09:00:46.327 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:00:46.327 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: debug -1> 2021-03-09T09:00:46.007+0000 7f00cfeb8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f00cfeb8700 time 2021-03-09T09:00:46.007450+0000 2021-03-09T09:00:46.328 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:46.328 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:46.328 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f00df23e1d8] 2021-03-09T09:00:46.328 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f00df23e3f2] 2021-03-09T09:00:46.329 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x557b40f3597c] 2021-03-09T09:00:46.329 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x557b40f74ced] 2021-03-09T09:00:46.329 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x557b40f9cb5a] 2021-03-09T09:00:46.329 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x557b40fa8054] 2021-03-09T09:00:46.329 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x557b40fac2c3] 2021-03-09T09:00:46.330 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x557b40f2dc2d] 2021-03-09T09:00:46.330 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x557b40e820b4] 2021-03-09T09:00:46.330 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 10: (C_MonOp::finish(int)+0x49) [0x557b40e21069] 2021-03-09T09:00:46.330 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 11: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.330 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x557b40e49da5] 2021-03-09T09:00:46.331 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 13: (Paxos::finish_round()+0x27b) [0x557b40f2553b] 2021-03-09T09:00:46.331 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 14: (Paxos::commit_finish()+0x8c0) [0x557b40f27f50] 2021-03-09T09:00:46.331 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 15: (C_Committed::finish(int)+0x45) [0x557b40f2bb85] 2021-03-09T09:00:46.331 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 16: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.332 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x557b40f2b8a8] 2021-03-09T09:00:46.332 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 18: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.332 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f00df2ddaa5] 2021-03-09T09:00:46.332 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 20: (Thread::_entry_func(void*)+0xd) [0x7f00df32541d] 2021-03-09T09:00:46.333 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f00dcd2414a] 2021-03-09T09:00:46.333 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 22: clone() 2021-03-09T09:00:46.333 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: debug 0> 2021-03-09T09:00:46.011+0000 7f00cfeb8700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:00:46.333 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: in thread 7f00cfeb8700 thread_name:fn_monstore 2021-03-09T09:00:46.333 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:46.334 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f00dcd2eb20] 2021-03-09T09:00:46.334 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 2: gsignal() 2021-03-09T09:00:46.334 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 3: abort() 2021-03-09T09:00:46.334 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f00df23e229] 2021-03-09T09:00:46.334 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f00df23e3f2] 2021-03-09T09:00:46.335 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x557b40f3597c] 2021-03-09T09:00:46.335 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x557b40f74ced] 2021-03-09T09:00:46.335 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x557b40f9cb5a] 2021-03-09T09:00:46.335 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x557b40fa8054] 2021-03-09T09:00:46.336 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x557b40fac2c3] 2021-03-09T09:00:46.336 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x557b40f2dc2d] 2021-03-09T09:00:46.336 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x557b40e820b4] 2021-03-09T09:00:46.336 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 13: (C_MonOp::finish(int)+0x49) [0x557b40e21069] 2021-03-09T09:00:46.336 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 14: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.337 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x557b40e49da5] 2021-03-09T09:00:46.337 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 16: (Paxos::finish_round()+0x27b) [0x557b40f2553b] 2021-03-09T09:00:46.337 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 17: (Paxos::commit_finish()+0x8c0) [0x557b40f27f50] 2021-03-09T09:00:46.337 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 18: (C_Committed::finish(int)+0x45) [0x557b40f2bb85] 2021-03-09T09:00:46.338 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 19: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.338 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x557b40f2b8a8] 2021-03-09T09:00:46.338 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 21: (Context::complete(int)+0xd) [0x557b40e1e49d] 2021-03-09T09:00:46.338 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f00df2ddaa5] 2021-03-09T09:00:46.338 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 23: (Thread::_entry_func(void*)+0xd) [0x7f00df32541d] 2021-03-09T09:00:46.339 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f00dcd2414a] 2021-03-09T09:00:46.339 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: 25: clone() 2021-03-09T09:00:46.339 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:46 smithi005 bash[25598]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:00:47.365 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:47 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Main process exited, code=exited, status=134/n/a 2021-03-09T09:00:47.365 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:47 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Failed with result 'exit-code'. 2021-03-09T09:00:47.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.091+0000 7f734891d700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-09T09:00:47.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.091+0000 7f734891d700 0 ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev), process ceph-mon, pid 6 2021-03-09T09:00:47.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.091+0000 7f734891d700 0 pidfile_write: ignore empty --pid-file 2021-03-09T09:00:47.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 0 load: jerasure load: lrc load: isa 2021-03-09T09:00:47.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: RocksDB version: 6.8.1 2021-03-09T09:00:47.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-09T09:00:47.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Compile date Mar 9 2021 2021-03-09T09:00:47.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: DB SUMMARY 2021-03-09T09:00:47.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: CURRENT file: CURRENT 2021-03-09T09:00:47.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-09T09:00:47.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: MANIFEST file: MANIFEST-000011 size: 275 Bytes 2021-03-09T09:00:47.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 3, files: 000004.sst 000007.sst 000010.sst 2021-03-09T09:00:47.373 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000012.log size: 18677 ; 2021-03-09T09:00:47.373 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.error_if_exists: 0 2021-03-09T09:00:47.374 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.create_if_missing: 0 2021-03-09T09:00:47.374 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.paranoid_checks: 1 2021-03-09T09:00:47.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.env: 0x5561205061c0 2021-03-09T09:00:47.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.fs: Posix File System 2021-03-09T09:00:47.376 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.info_log: 0x55612a5b7120 2021-03-09T09:00:47.376 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-09T09:00:47.377 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.statistics: (nil) 2021-03-09T09:00:47.377 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.use_fsync: 0 2021-03-09T09:00:47.378 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_log_file_size: 0 2021-03-09T09:00:47.378 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-09T09:00:47.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-09T09:00:47.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-09T09:00:47.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-09T09:00:47.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.allow_fallocate: 1 2021-03-09T09:00:47.381 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-09T09:00:47.381 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-09T09:00:47.382 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.use_direct_reads: 0 2021-03-09T09:00:47.382 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-09T09:00:47.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-09T09:00:47.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.db_log_dir: 2021-03-09T09:00:47.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2021-03-09T09:00:47.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-09T09:00:47.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_subcompactions: 1 2021-03-09T09:00:47.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_background_flushes: -1 2021-03-09T09:00:47.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-09T09:00:47.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-09T09:00:47.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-09T09:00:47.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-09T09:00:47.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-09T09:00:47.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-09T09:00:47.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-09T09:00:47.389 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.write_buffer_manager: 0x55612a60bcb0 2021-03-09T09:00:47.389 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-09T09:00:47.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-09T09:00:47.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-09T09:00:47.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-09T09:00:47.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-09T09:00:47.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-09T09:00:47.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-09T09:00:47.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-09T09:00:47.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-09T09:00:47.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.unordered_write: 0 2021-03-09T09:00:47.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-09T09:00:47.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-09T09:00:47.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-09T09:00:47.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-09T09:00:47.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.row_cache: None 2021-03-09T09:00:47.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.wal_filter: None 2021-03-09T09:00:47.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-09T09:00:47.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-09T09:00:47.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.preserve_deletes: 0 2021-03-09T09:00:47.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.two_write_queues: 0 2021-03-09T09:00:47.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-09T09:00:47.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.atomic_flush: 0 2021-03-09T09:00:47.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-09T09:00:47.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-09T09:00:47.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-09T09:00:47.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.log_readahead_size: 0 2021-03-09T09:00:47.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-09T09:00:47.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_background_jobs: 2 2021-03-09T09:00:47.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_background_compactions: -1 2021-03-09T09:00:47.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-09T09:00:47.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-09T09:00:47.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-09T09:00:47.408 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-09T09:00:47.408 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-09T09:00:47.408 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-09T09:00:47.409 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-09T09:00:47.409 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-09T09:00:47.410 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_open_files: -1 2021-03-09T09:00:47.410 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-09T09:00:47.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-09T09:00:47.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-09T09:00:47.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-09T09:00:47.412 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Compression algorithms supported: 2021-03-09T09:00:47.412 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-09T09:00:47.413 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: kZSTD supported: 0 2021-03-09T09:00:47.413 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: kXpressCompression supported: 0 2021-03-09T09:00:47.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-09T09:00:47.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: kLZ4Compression supported: 1 2021-03-09T09:00:47.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: kBZip2Compression supported: 0 2021-03-09T09:00:47.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: kZlibCompression supported: 1 2021-03-09T09:00:47.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: kSnappyCompression supported: 1 2021-03-09T09:00:47.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-09T09:00:47.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000011 2021-03-09T09:00:47.417 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-09T09:00:47.417 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-09T09:00:47.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.merge_operator: 2021-03-09T09:00:47.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compaction_filter: None 2021-03-09T09:00:47.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-09T09:00:47.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-09T09:00:47.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-09T09:00:47.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55612235a230) 2021-03-09T09:00:47.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: cache_index_and_filter_blocks: 1 2021-03-09T09:00:47.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-09T09:00:47.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-09T09:00:47.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: pin_top_level_index_and_filter: 1 2021-03-09T09:00:47.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: index_type: 0 2021-03-09T09:00:47.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: data_block_index_type: 0 2021-03-09T09:00:47.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: index_shortening: 1 2021-03-09T09:00:47.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: data_block_hash_table_util_ratio: 0.750000 2021-03-09T09:00:47.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: hash_index_allow_collision: 1 2021-03-09T09:00:47.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: checksum: 1 2021-03-09T09:00:47.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: no_block_cache: 0 2021-03-09T09:00:47.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: block_cache: 0x55612a592510 2021-03-09T09:00:47.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: block_cache_name: BinnedLRUCache 2021-03-09T09:00:47.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: block_cache_options: 2021-03-09T09:00:47.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: capacity : 536870912 2021-03-09T09:00:47.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: num_shard_bits : 4 2021-03-09T09:00:47.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: strict_capacity_limit : 0 2021-03-09T09:00:47.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: high_pri_pool_ratio: 0.000 2021-03-09T09:00:47.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: block_cache_compressed: (nil) 2021-03-09T09:00:47.429 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: persistent_cache: (nil) 2021-03-09T09:00:47.429 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: block_size: 4096 2021-03-09T09:00:47.430 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: block_size_deviation: 10 2021-03-09T09:00:47.430 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: block_restart_interval: 16 2021-03-09T09:00:47.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: index_block_restart_interval: 1 2021-03-09T09:00:47.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: metadata_block_size: 4096 2021-03-09T09:00:47.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: partition_filters: 0 2021-03-09T09:00:47.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: use_delta_encoding: 1 2021-03-09T09:00:47.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-09T09:00:47.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: whole_key_filtering: 1 2021-03-09T09:00:47.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: verify_compression: 0 2021-03-09T09:00:47.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: read_amp_bytes_per_bit: 0 2021-03-09T09:00:47.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: format_version: 2 2021-03-09T09:00:47.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: enable_index_compression: 1 2021-03-09T09:00:47.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: block_align: 0 2021-03-09T09:00:47.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-09T09:00:47.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-09T09:00:47.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compression: NoCompression 2021-03-09T09:00:47.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-09T09:00:47.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-09T09:00:47.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-09T09:00:47.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.num_levels: 7 2021-03-09T09:00:47.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-09T09:00:47.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-09T09:00:47.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-09T09:00:47.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-09T09:00:47.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-09T09:00:47.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-09T09:00:47.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-09T09:00:47.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:00:47.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-09T09:00:47.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-09T09:00:47.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-09T09:00:47.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-09T09:00:47.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-09T09:00:47.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:00:47.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-09T09:00:47.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-09T09:00:47.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-09T09:00:47.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-09T09:00:47.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-09T09:00:47.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-09T09:00:47.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-09T09:00:47.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-09T09:00:47.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-09T09:00:47.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-09T09:00:47.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-09T09:00:47.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-09T09:00:47.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-09T09:00:47.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-09T09:00:47.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-09T09:00:47.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-09T09:00:47.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-09T09:00:47.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-09T09:00:47.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-09T09:00:47.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-09T09:00:47.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-09T09:00:47.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-09T09:00:47.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-09T09:00:47.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-09T09:00:47.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-09T09:00:47.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-09T09:00:47.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-09T09:00:47.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-09T09:00:47.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-09T09:00:47.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-09T09:00:47.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-09T09:00:47.450 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-09T09:00:47.450 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-09T09:00:47.450 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.table_properties_collectors: 2021-03-09T09:00:47.450 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.inplace_update_support: 0 2021-03-09T09:00:47.451 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-09T09:00:47.451 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-09T09:00:47.451 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-09T09:00:47.451 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-09T09:00:47.451 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.bloom_locality: 0 2021-03-09T09:00:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.max_successive_merges: 0 2021-03-09T09:00:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-09T09:00:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-09T09:00:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-09T09:00:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-09T09:00:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.ttl: 2592000 2021-03-09T09:00:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.095+0000 7f734891d700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-09T09:00:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.099+0000 7f734891d700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000011 succeeded,manifest_file_number is 11, next_file_number is 13, last_sequence is 3228, log_number is 10,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-09T09:00:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.099+0000 7f734891d700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 10 2021-03-09T09:00:47.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.099+0000 7f734891d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280447101745, "job": 1, "event": "recovery_started", "log_files": [12]} 2021-03-09T09:00:47.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.099+0000 7f734891d700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #12 mode 2 2021-03-09T09:00:47.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.099+0000 7f734891d700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-09T09:00:47.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.099+0000 7f734891d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280447102841, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 15142, "table_properties": {"data_size": 14177, "index_size": 90, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 447, "raw_average_key_size": 23, "raw_value_size": 13733, "raw_average_value_size": 722, "num_data_blocks": 3, "num_entries": 19, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1615280447, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-09T09:00:47.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.099+0000 7f734891d700 4 rocksdb: [version_set.cc:3826] Creating manifest 14 2021-03-09T09:00:47.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.099+0000 7f734891d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280447103458, "job": 1, "event": "recovery_finished"} 2021-03-09T09:00:47.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.103+0000 7f734891d700 4 rocksdb: DB pointer 0x55612a657800 2021-03-09T09:00:47.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.103+0000 7f734013e700 4 rocksdb: [compaction/compaction_job.cc:1676] [default] [JOB 3] Compacting 4@0 files to L6, score 1.00 2021-03-09T09:00:47.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.103+0000 7f734013e700 4 rocksdb: [compaction/compaction_job.cc:1680] [default] Compaction start summary: Base version 2 Base level 0, inputs: [13(14KB) 10(7769KB) 7(80KB) 4(1694B)] 2021-03-09T09:00:47.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.103+0000 7f734013e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280447105297, "job": 3, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [13, 10, 7, 4], "score": 1, "input_data_size": 8055032} 2021-03-09T09:00:47.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.103+0000 7f733792d700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-09T09:00:47.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: debug 2021-03-09T09:00:47.103+0000 7f733792d700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-09T09:00:47.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: ** DB Stats ** 2021-03-09T09:00:47.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:00:47.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-09T09:00:47.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-09T09:00:47.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:00:47.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-09T09:00:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-09T09:00:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:00:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: ** Compaction Stats [default] ** 2021-03-09T09:00:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: L0 4/4 7.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 16.6 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Sum 4/4 7.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 16.6 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 16.6 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: ** Compaction Stats [default] ** 2021-03-09T09:00:47.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:47.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:47.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 16.6 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:47.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:00:47.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T09:00:47.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:00:47.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:00:47.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:00:47.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:00:47.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Cumulative compaction: 0.00 GB write, 2.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:47.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Interval compaction: 0.00 GB write, 2.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:47.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:00:47.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:00:47.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: ** Compaction Stats [default] ** 2021-03-09T09:00:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: L0 4/4 7.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 16.6 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Sum 4/4 7.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 16.6 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:00:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: ** Compaction Stats [default] ** 2021-03-09T09:00:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 16.6 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:47.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:00:47.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T09:00:47.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:00:47.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:00:47.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:00:47.466 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:00:47.466 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Cumulative compaction: 0.00 GB write, 2.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:47.466 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:47.466 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:00:47.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:00:47.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: audit 2021-03-09T08:59:48.920210+0000 mon.a (mon.0) 269 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T09:00:47.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: cluster 2021-03-09T08:59:48.920524+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-09T09:00:47.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: audit 2021-03-09T08:59:48.921424+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:00:47.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:47 smithi005 bash[25708]: audit 2021-03-09T08:59:48.930506+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:00:49.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:00:49 smithi146 bash[13136]: debug 2021-03-09T09:00:49.358+0000 7fa560337700 -1 mon.b@2(peon) e3 get_health_metrics reporting 38 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:00:54.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:00:54 smithi146 bash[13136]: debug 2021-03-09T09:00:54.358+0000 7fa560337700 -1 mon.b@2(electing) e3 get_health_metrics reporting 48 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:00:56.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f733892f700 time 2021-03-09T09:00:55.654632+0000 2021-03-09T09:00:56.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:56.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:56.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f7347cb51d8] 2021-03-09T09:00:56.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f7347cb53f2] 2021-03-09T09:00:56.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55611fd6897c] 2021-03-09T09:00:56.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55611fda7ced] 2021-03-09T09:00:56.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55611fdcfb5a] 2021-03-09T09:00:56.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55611fddb054] 2021-03-09T09:00:56.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55611fddf2c3] 2021-03-09T09:00:56.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55611fd60c2d] 2021-03-09T09:00:56.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55611fcb50b4] 2021-03-09T09:00:56.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 10: (C_MonOp::finish(int)+0x49) [0x55611fc54069] 2021-03-09T09:00:56.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 11: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55611fc7cda5] 2021-03-09T09:00:56.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 13: (Paxos::finish_round()+0x27b) [0x55611fd5853b] 2021-03-09T09:00:56.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 14: (Paxos::commit_finish()+0x8c0) [0x55611fd5af50] 2021-03-09T09:00:56.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 15: (C_Committed::finish(int)+0x45) [0x55611fd5eb85] 2021-03-09T09:00:56.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 16: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55611fd5e8a8] 2021-03-09T09:00:56.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 18: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f7347d54aa5] 2021-03-09T09:00:56.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 20: (Thread::_entry_func(void*)+0xd) [0x7f7347d9c41d] 2021-03-09T09:00:56.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f734579b14a] 2021-03-09T09:00:56.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 22: clone() 2021-03-09T09:00:56.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: *** Caught signal (Aborted) ** 2021-03-09T09:00:56.121 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: in thread 7f733892f700 thread_name:fn_monstore 2021-03-09T09:00:56.121 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: debug 2021-03-09T09:00:55.659+0000 7f733892f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f733892f700 time 2021-03-09T09:00:55.654632+0000 2021-03-09T09:00:56.121 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:56.121 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:56.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f7347cb51d8] 2021-03-09T09:00:56.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f7347cb53f2] 2021-03-09T09:00:56.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55611fd6897c] 2021-03-09T09:00:56.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55611fda7ced] 2021-03-09T09:00:56.123 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55611fdcfb5a] 2021-03-09T09:00:56.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55611fddb054] 2021-03-09T09:00:56.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55611fddf2c3] 2021-03-09T09:00:56.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55611fd60c2d] 2021-03-09T09:00:56.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55611fcb50b4] 2021-03-09T09:00:56.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 10: (C_MonOp::finish(int)+0x49) [0x55611fc54069] 2021-03-09T09:00:56.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 11: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55611fc7cda5] 2021-03-09T09:00:56.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 13: (Paxos::finish_round()+0x27b) [0x55611fd5853b] 2021-03-09T09:00:56.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 14: (Paxos::commit_finish()+0x8c0) [0x55611fd5af50] 2021-03-09T09:00:56.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 15: (C_Committed::finish(int)+0x45) [0x55611fd5eb85] 2021-03-09T09:00:56.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 16: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55611fd5e8a8] 2021-03-09T09:00:56.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 18: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f7347d54aa5] 2021-03-09T09:00:56.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 20: (Thread::_entry_func(void*)+0xd) [0x7f7347d9c41d] 2021-03-09T09:00:56.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f734579b14a] 2021-03-09T09:00:56.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 22: clone() 2021-03-09T09:00:56.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:56.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f73457a5b20] 2021-03-09T09:00:56.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 2: gsignal() 2021-03-09T09:00:56.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 3: abort() 2021-03-09T09:00:56.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f7347cb5229] 2021-03-09T09:00:56.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f7347cb53f2] 2021-03-09T09:00:56.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55611fd6897c] 2021-03-09T09:00:56.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55611fda7ced] 2021-03-09T09:00:56.130 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55611fdcfb5a] 2021-03-09T09:00:56.130 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55611fddb054] 2021-03-09T09:00:56.130 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55611fddf2c3] 2021-03-09T09:00:56.130 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55611fd60c2d] 2021-03-09T09:00:56.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55611fcb50b4] 2021-03-09T09:00:56.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 13: (C_MonOp::finish(int)+0x49) [0x55611fc54069] 2021-03-09T09:00:56.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 14: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55611fc7cda5] 2021-03-09T09:00:56.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 16: (Paxos::finish_round()+0x27b) [0x55611fd5853b] 2021-03-09T09:00:56.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 17: (Paxos::commit_finish()+0x8c0) [0x55611fd5af50] 2021-03-09T09:00:56.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 18: (C_Committed::finish(int)+0x45) [0x55611fd5eb85] 2021-03-09T09:00:56.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 19: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55611fd5e8a8] 2021-03-09T09:00:56.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 21: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f7347d54aa5] 2021-03-09T09:00:56.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 23: (Thread::_entry_func(void*)+0xd) [0x7f7347d9c41d] 2021-03-09T09:00:56.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f734579b14a] 2021-03-09T09:00:56.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 25: clone() 2021-03-09T09:00:56.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: debug 2021-03-09T09:00:55.663+0000 7f733892f700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:00:56.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: in thread 7f733892f700 thread_name:fn_monstore 2021-03-09T09:00:56.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:56.135 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f73457a5b20] 2021-03-09T09:00:56.135 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 2: gsignal() 2021-03-09T09:00:56.135 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 3: abort() 2021-03-09T09:00:56.135 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f7347cb5229] 2021-03-09T09:00:56.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f7347cb53f2] 2021-03-09T09:00:56.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55611fd6897c] 2021-03-09T09:00:56.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55611fda7ced] 2021-03-09T09:00:56.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55611fdcfb5a] 2021-03-09T09:00:56.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55611fddb054] 2021-03-09T09:00:56.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55611fddf2c3] 2021-03-09T09:00:56.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55611fd60c2d] 2021-03-09T09:00:56.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55611fcb50b4] 2021-03-09T09:00:56.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 13: (C_MonOp::finish(int)+0x49) [0x55611fc54069] 2021-03-09T09:00:56.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 14: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55611fc7cda5] 2021-03-09T09:00:56.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 16: (Paxos::finish_round()+0x27b) [0x55611fd5853b] 2021-03-09T09:00:56.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 17: (Paxos::commit_finish()+0x8c0) [0x55611fd5af50] 2021-03-09T09:00:56.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 18: (C_Committed::finish(int)+0x45) [0x55611fd5eb85] 2021-03-09T09:00:56.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 19: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55611fd5e8a8] 2021-03-09T09:00:56.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 21: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f7347d54aa5] 2021-03-09T09:00:56.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 23: (Thread::_entry_func(void*)+0xd) [0x7f7347d9c41d] 2021-03-09T09:00:56.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f734579b14a] 2021-03-09T09:00:56.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 25: clone() 2021-03-09T09:00:56.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:00:56.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: debug -1> 2021-03-09T09:00:55.659+0000 7f733892f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f733892f700 time 2021-03-09T09:00:55.654632+0000 2021-03-09T09:00:56.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:56.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:56.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f7347cb51d8] 2021-03-09T09:00:56.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f7347cb53f2] 2021-03-09T09:00:56.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55611fd6897c] 2021-03-09T09:00:56.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55611fda7ced] 2021-03-09T09:00:56.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55611fdcfb5a] 2021-03-09T09:00:56.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55611fddb054] 2021-03-09T09:00:56.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55611fddf2c3] 2021-03-09T09:00:56.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55611fd60c2d] 2021-03-09T09:00:56.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55611fcb50b4] 2021-03-09T09:00:56.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 10: (C_MonOp::finish(int)+0x49) [0x55611fc54069] 2021-03-09T09:00:56.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 11: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55611fc7cda5] 2021-03-09T09:00:56.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 13: (Paxos::finish_round()+0x27b) [0x55611fd5853b] 2021-03-09T09:00:56.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 14: (Paxos::commit_finish()+0x8c0) [0x55611fd5af50] 2021-03-09T09:00:56.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 15: (C_Committed::finish(int)+0x45) [0x55611fd5eb85] 2021-03-09T09:00:56.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 16: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55611fd5e8a8] 2021-03-09T09:00:56.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 18: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f7347d54aa5] 2021-03-09T09:00:56.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 20: (Thread::_entry_func(void*)+0xd) [0x7f7347d9c41d] 2021-03-09T09:00:56.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f734579b14a] 2021-03-09T09:00:56.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 22: clone() 2021-03-09T09:00:56.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: debug 0> 2021-03-09T09:00:55.663+0000 7f733892f700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:00:56.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: in thread 7f733892f700 thread_name:fn_monstore 2021-03-09T09:00:56.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:56.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f73457a5b20] 2021-03-09T09:00:56.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 2: gsignal() 2021-03-09T09:00:56.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 3: abort() 2021-03-09T09:00:56.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f7347cb5229] 2021-03-09T09:00:56.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f7347cb53f2] 2021-03-09T09:00:56.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55611fd6897c] 2021-03-09T09:00:56.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55611fda7ced] 2021-03-09T09:00:56.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55611fdcfb5a] 2021-03-09T09:00:56.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55611fddb054] 2021-03-09T09:00:56.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55611fddf2c3] 2021-03-09T09:00:56.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55611fd60c2d] 2021-03-09T09:00:56.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55611fcb50b4] 2021-03-09T09:00:56.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 13: (C_MonOp::finish(int)+0x49) [0x55611fc54069] 2021-03-09T09:00:56.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 14: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55611fc7cda5] 2021-03-09T09:00:56.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 16: (Paxos::finish_round()+0x27b) [0x55611fd5853b] 2021-03-09T09:00:56.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 17: (Paxos::commit_finish()+0x8c0) [0x55611fd5af50] 2021-03-09T09:00:56.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 18: (C_Committed::finish(int)+0x45) [0x55611fd5eb85] 2021-03-09T09:00:56.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 19: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55611fd5e8a8] 2021-03-09T09:00:56.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 21: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f7347d54aa5] 2021-03-09T09:00:56.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 23: (Thread::_entry_func(void*)+0xd) [0x7f7347d9c41d] 2021-03-09T09:00:56.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f734579b14a] 2021-03-09T09:00:56.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 25: clone() 2021-03-09T09:00:56.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:00:56.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: debug -1> 2021-03-09T09:00:55.659+0000 7f733892f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f733892f700 time 2021-03-09T09:00:55.654632+0000 2021-03-09T09:00:56.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:00:56.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:56.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f7347cb51d8] 2021-03-09T09:00:56.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f7347cb53f2] 2021-03-09T09:00:56.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55611fd6897c] 2021-03-09T09:00:56.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55611fda7ced] 2021-03-09T09:00:56.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55611fdcfb5a] 2021-03-09T09:00:56.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55611fddb054] 2021-03-09T09:00:56.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55611fddf2c3] 2021-03-09T09:00:56.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55611fd60c2d] 2021-03-09T09:00:56.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55611fcb50b4] 2021-03-09T09:00:56.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 10: (C_MonOp::finish(int)+0x49) [0x55611fc54069] 2021-03-09T09:00:56.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 11: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55611fc7cda5] 2021-03-09T09:00:56.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 13: (Paxos::finish_round()+0x27b) [0x55611fd5853b] 2021-03-09T09:00:56.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 14: (Paxos::commit_finish()+0x8c0) [0x55611fd5af50] 2021-03-09T09:00:56.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 15: (C_Committed::finish(int)+0x45) [0x55611fd5eb85] 2021-03-09T09:00:56.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 16: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55611fd5e8a8] 2021-03-09T09:00:56.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 18: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f7347d54aa5] 2021-03-09T09:00:56.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 20: (Thread::_entry_func(void*)+0xd) [0x7f7347d9c41d] 2021-03-09T09:00:56.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f734579b14a] 2021-03-09T09:00:56.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 22: clone() 2021-03-09T09:00:56.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: debug 0> 2021-03-09T09:00:55.663+0000 7f733892f700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:00:56.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: in thread 7f733892f700 thread_name:fn_monstore 2021-03-09T09:00:56.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:00:56.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f73457a5b20] 2021-03-09T09:00:56.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 2: gsignal() 2021-03-09T09:00:56.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 3: abort() 2021-03-09T09:00:56.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f7347cb5229] 2021-03-09T09:00:56.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f7347cb53f2] 2021-03-09T09:00:56.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55611fd6897c] 2021-03-09T09:00:56.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55611fda7ced] 2021-03-09T09:00:56.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55611fdcfb5a] 2021-03-09T09:00:56.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55611fddb054] 2021-03-09T09:00:56.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55611fddf2c3] 2021-03-09T09:00:56.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55611fd60c2d] 2021-03-09T09:00:56.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55611fcb50b4] 2021-03-09T09:00:56.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 13: (C_MonOp::finish(int)+0x49) [0x55611fc54069] 2021-03-09T09:00:56.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 14: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55611fc7cda5] 2021-03-09T09:00:56.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 16: (Paxos::finish_round()+0x27b) [0x55611fd5853b] 2021-03-09T09:00:56.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 17: (Paxos::commit_finish()+0x8c0) [0x55611fd5af50] 2021-03-09T09:00:56.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 18: (C_Committed::finish(int)+0x45) [0x55611fd5eb85] 2021-03-09T09:00:56.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 19: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55611fd5e8a8] 2021-03-09T09:00:56.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 21: (Context::complete(int)+0xd) [0x55611fc5149d] 2021-03-09T09:00:56.165 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f7347d54aa5] 2021-03-09T09:00:56.165 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 23: (Thread::_entry_func(void*)+0xd) [0x7f7347d9c41d] 2021-03-09T09:00:56.165 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f734579b14a] 2021-03-09T09:00:56.165 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: 25: clone() 2021-03-09T09:00:56.166 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:55 smithi005 bash[25708]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:00:56.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:56 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Main process exited, code=exited, status=134/n/a 2021-03-09T09:00:56.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:00:56 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Failed with result 'exit-code'. 2021-03-09T09:00:57.614 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:57 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Scheduled restart job, restart counter is at 3. 2021-03-09T09:00:57.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:57 smithi005 systemd[1]: Stopped Ceph mon.c for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:00:57.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:57 smithi005 systemd[1]: Started Ceph mon.c for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:00:58.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.299+0000 7f9c7f9c3700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-09T09:00:58.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.299+0000 7f9c7f9c3700 0 ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev), process ceph-mon, pid 6 2021-03-09T09:00:58.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.299+0000 7f9c7f9c3700 0 pidfile_write: ignore empty --pid-file 2021-03-09T09:00:58.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 0 load: jerasure load: lrc load: isa 2021-03-09T09:00:58.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: RocksDB version: 6.8.1 2021-03-09T09:00:58.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-09T09:00:58.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Compile date Mar 9 2021 2021-03-09T09:00:58.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: DB SUMMARY 2021-03-09T09:00:58.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: CURRENT file: CURRENT 2021-03-09T09:00:58.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-09T09:00:58.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: MANIFEST file: MANIFEST-000011 size: 280 Bytes 2021-03-09T09:00:58.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 3, files: 000004.sst 000007.sst 000010.sst 2021-03-09T09:00:58.621 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000012.log size: 18634 ; 2021-03-09T09:00:58.621 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.error_if_exists: 0 2021-03-09T09:00:58.622 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.create_if_missing: 0 2021-03-09T09:00:58.622 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.paranoid_checks: 1 2021-03-09T09:00:58.622 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.env: 0x560d296b71c0 2021-03-09T09:00:58.623 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.fs: Posix File System 2021-03-09T09:00:58.623 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.info_log: 0x560d327d5120 2021-03-09T09:00:58.624 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-09T09:00:58.624 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.statistics: (nil) 2021-03-09T09:00:58.625 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.use_fsync: 0 2021-03-09T09:00:58.625 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_log_file_size: 0 2021-03-09T09:00:58.626 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-09T09:00:58.626 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-09T09:00:58.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-09T09:00:58.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-09T09:00:58.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.allow_fallocate: 1 2021-03-09T09:00:58.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-09T09:00:58.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-09T09:00:58.629 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.use_direct_reads: 0 2021-03-09T09:00:58.629 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-09T09:00:58.630 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-09T09:00:58.630 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.db_log_dir: 2021-03-09T09:00:58.630 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2021-03-09T09:00:58.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-09T09:00:58.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_subcompactions: 1 2021-03-09T09:00:58.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_background_flushes: -1 2021-03-09T09:00:58.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-09T09:00:58.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-09T09:00:58.633 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-09T09:00:58.633 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-09T09:00:58.634 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-09T09:00:58.634 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-09T09:00:58.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-09T09:00:58.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.write_buffer_manager: 0x560d32829ce0 2021-03-09T09:00:58.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-09T09:00:58.636 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-09T09:00:58.636 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-09T09:00:58.637 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-09T09:00:58.637 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-09T09:00:58.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-09T09:00:58.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-09T09:00:58.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-09T09:00:58.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-09T09:00:58.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.unordered_write: 0 2021-03-09T09:00:58.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-09T09:00:58.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-09T09:00:58.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-09T09:00:58.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-09T09:00:58.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.row_cache: None 2021-03-09T09:00:58.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.wal_filter: None 2021-03-09T09:00:58.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-09T09:00:58.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-09T09:00:58.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.preserve_deletes: 0 2021-03-09T09:00:58.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.two_write_queues: 0 2021-03-09T09:00:58.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-09T09:00:58.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.atomic_flush: 0 2021-03-09T09:00:58.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-09T09:00:58.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-09T09:00:58.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-09T09:00:58.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.log_readahead_size: 0 2021-03-09T09:00:58.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-09T09:00:58.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_background_jobs: 2 2021-03-09T09:00:58.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_background_compactions: -1 2021-03-09T09:00:58.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-09T09:00:58.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-09T09:00:58.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-09T09:00:58.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-09T09:00:58.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-09T09:00:58.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-09T09:00:58.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-09T09:00:58.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-09T09:00:58.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_open_files: -1 2021-03-09T09:00:58.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-09T09:00:58.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-09T09:00:58.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-09T09:00:58.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-09T09:00:58.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Compression algorithms supported: 2021-03-09T09:00:58.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-09T09:00:58.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: kZSTD supported: 0 2021-03-09T09:00:58.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: kXpressCompression supported: 0 2021-03-09T09:00:58.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-09T09:00:58.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: kLZ4Compression supported: 1 2021-03-09T09:00:58.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: kBZip2Compression supported: 0 2021-03-09T09:00:58.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: kZlibCompression supported: 1 2021-03-09T09:00:58.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: kSnappyCompression supported: 1 2021-03-09T09:00:58.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-09T09:00:58.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000011 2021-03-09T09:00:58.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-09T09:00:58.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-09T09:00:58.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.merge_operator: 2021-03-09T09:00:58.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compaction_filter: None 2021-03-09T09:00:58.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-09T09:00:58.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-09T09:00:58.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-09T09:00:58.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560d2a578230) 2021-03-09T09:00:58.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: cache_index_and_filter_blocks: 1 2021-03-09T09:00:58.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-09T09:00:58.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-09T09:00:58.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: pin_top_level_index_and_filter: 1 2021-03-09T09:00:58.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: index_type: 0 2021-03-09T09:00:58.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: data_block_index_type: 0 2021-03-09T09:00:58.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: index_shortening: 1 2021-03-09T09:00:58.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: data_block_hash_table_util_ratio: 0.750000 2021-03-09T09:00:58.658 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: hash_index_allow_collision: 1 2021-03-09T09:00:58.658 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: checksum: 1 2021-03-09T09:00:58.658 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: no_block_cache: 0 2021-03-09T09:00:58.658 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: block_cache: 0x560d327b0510 2021-03-09T09:00:58.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: block_cache_name: BinnedLRUCache 2021-03-09T09:00:58.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: block_cache_options: 2021-03-09T09:00:58.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: capacity : 536870912 2021-03-09T09:00:58.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: num_shard_bits : 4 2021-03-09T09:00:58.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: strict_capacity_limit : 0 2021-03-09T09:00:58.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: high_pri_pool_ratio: 0.000 2021-03-09T09:00:58.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: block_cache_compressed: (nil) 2021-03-09T09:00:58.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: persistent_cache: (nil) 2021-03-09T09:00:58.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: block_size: 4096 2021-03-09T09:00:58.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: block_size_deviation: 10 2021-03-09T09:00:58.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: block_restart_interval: 16 2021-03-09T09:00:58.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: index_block_restart_interval: 1 2021-03-09T09:00:58.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: metadata_block_size: 4096 2021-03-09T09:00:58.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: partition_filters: 0 2021-03-09T09:00:58.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: use_delta_encoding: 1 2021-03-09T09:00:58.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-09T09:00:58.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: whole_key_filtering: 1 2021-03-09T09:00:58.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: verify_compression: 0 2021-03-09T09:00:58.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: read_amp_bytes_per_bit: 0 2021-03-09T09:00:58.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: format_version: 2 2021-03-09T09:00:58.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: enable_index_compression: 1 2021-03-09T09:00:58.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: block_align: 0 2021-03-09T09:00:58.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-09T09:00:58.664 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-09T09:00:58.664 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compression: NoCompression 2021-03-09T09:00:58.664 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-09T09:00:58.664 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-09T09:00:58.665 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-09T09:00:58.665 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.num_levels: 7 2021-03-09T09:00:58.665 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-09T09:00:58.665 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-09T09:00:58.665 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-09T09:00:58.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-09T09:00:58.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-09T09:00:58.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-09T09:00:58.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-09T09:00:58.667 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:00:58.667 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-09T09:00:58.667 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-09T09:00:58.667 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-09T09:00:58.667 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-09T09:00:58.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-09T09:00:58.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:00:58.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-09T09:00:58.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-09T09:00:58.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-09T09:00:58.669 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-09T09:00:58.669 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-09T09:00:58.669 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-09T09:00:58.669 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-09T09:00:58.670 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-09T09:00:58.670 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-09T09:00:58.670 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-09T09:00:58.670 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-09T09:00:58.670 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-09T09:00:58.671 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-09T09:00:58.671 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-09T09:00:58.671 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-09T09:00:58.671 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-09T09:00:58.671 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-09T09:00:58.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-09T09:00:58.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-09T09:00:58.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-09T09:00:58.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-09T09:00:58.673 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-09T09:00:58.673 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-09T09:00:58.673 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-09T09:00:58.673 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-09T09:00:58.673 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-09T09:00:58.674 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-09T09:00:58.674 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-09T09:00:58.674 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-09T09:00:58.674 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-09T09:00:58.674 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-09T09:00:58.675 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-09T09:00:58.675 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-09T09:00:58.675 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.table_properties_collectors: 2021-03-09T09:00:58.675 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.inplace_update_support: 0 2021-03-09T09:00:58.675 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-09T09:00:58.676 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-09T09:00:58.676 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-09T09:00:58.676 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-09T09:00:58.676 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.bloom_locality: 0 2021-03-09T09:00:58.677 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.max_successive_merges: 0 2021-03-09T09:00:58.677 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-09T09:00:58.677 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-09T09:00:58.677 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-09T09:00:58.677 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-09T09:00:58.678 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.ttl: 2592000 2021-03-09T09:00:58.678 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-09T09:00:58.678 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000011 succeeded,manifest_file_number is 11, next_file_number is 13, last_sequence is 2941, log_number is 10,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-09T09:00:58.678 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 10 2021-03-09T09:00:58.678 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280458311373, "job": 1, "event": "recovery_started", "log_files": [12]} 2021-03-09T09:00:58.679 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #12 mode 2 2021-03-09T09:00:58.679 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.307+0000 7f9c7f9c3700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-09T09:00:58.679 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.311+0000 7f9c7f9c3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280458312538, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 15142, "table_properties": {"data_size": 14177, "index_size": 90, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 447, "raw_average_key_size": 23, "raw_value_size": 13733, "raw_average_value_size": 722, "num_data_blocks": 3, "num_entries": 19, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1615280458, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-09T09:00:58.679 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.311+0000 7f9c7f9c3700 4 rocksdb: [version_set.cc:3826] Creating manifest 14 2021-03-09T09:00:58.680 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.311+0000 7f9c7f9c3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280458313425, "job": 1, "event": "recovery_finished"} 2021-03-09T09:00:58.680 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.311+0000 7f9c7f9c3700 4 rocksdb: DB pointer 0x560d32875800 2021-03-09T09:00:58.680 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.311+0000 7f9c771e4700 4 rocksdb: [compaction/compaction_job.cc:1676] [default] [JOB 3] Compacting 4@0 files to L6, score 1.00 2021-03-09T09:00:58.680 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.311+0000 7f9c771e4700 4 rocksdb: [compaction/compaction_job.cc:1680] [default] Compaction start summary: Base version 2 Base level 0, inputs: [13(14KB) 10(11KB) 7(7842KB) 4(1437B)] 2021-03-09T09:00:58.680 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.311+0000 7f9c771e4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280458315216, "job": 3, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [13, 10, 7, 4], "score": 1, "input_data_size": 8058801} 2021-03-09T09:00:58.681 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.311+0000 7f9c6e9d3700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-09T09:00:58.681 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.311+0000 7f9c6e9d3700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-09T09:00:58.681 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: ** DB Stats ** 2021-03-09T09:00:58.681 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:00:58.682 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-09T09:00:58.682 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-09T09:00:58.682 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:00:58.682 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-09T09:00:58.683 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-09T09:00:58.683 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:00:58.683 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: ** Compaction Stats [default] ** 2021-03-09T09:00:58.683 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:58.683 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:58.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: L0 4/4 7.69 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 14.9 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:58.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Sum 4/4 7.69 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 14.9 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:58.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 14.9 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:58.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: ** Compaction Stats [default] ** 2021-03-09T09:00:58.685 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:58.685 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:58.685 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 14.9 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:58.685 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:00:58.686 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T09:00:58.686 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:00:58.686 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:00:58.686 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:00:58.686 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:00:58.687 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Cumulative compaction: 0.00 GB write, 2.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:58.687 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Interval compaction: 0.00 GB write, 2.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:58.687 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:00:58.687 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:00:58.688 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: ** Compaction Stats [default] ** 2021-03-09T09:00:58.688 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:58.688 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:58.688 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: L0 4/4 7.69 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 14.9 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:58.688 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Sum 4/4 7.69 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 14.9 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:58.689 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:00:58.689 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: ** Compaction Stats [default] ** 2021-03-09T09:00:58.689 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:00:58.689 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:00:58.689 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 14.9 0.00 0.00 1 0.001 0 0 2021-03-09T09:00:58.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:00:58.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T09:00:58.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:00:58.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:00:58.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:00:58.691 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:00:58.691 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Cumulative compaction: 0.00 GB write, 2.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:58.691 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:00:58.691 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:00:58.691 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:00:58.692 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.315+0000 7f9c7f9c3700 0 starting mon.c rank 1 at public addrs [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] at bind addrs [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:00:58.692 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.315+0000 7f9c7f9c3700 1 mon.c@-1(???) e3 preinit fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:00:58.692 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.315+0000 7f9c7f9c3700 0 mon.c@-1(???).mds e1 new map 2021-03-09T09:00:58.692 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.315+0000 7f9c7f9c3700 0 mon.c@-1(???).mds e1 print_map 2021-03-09T09:00:58.693 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: e1 2021-03-09T09:00:58.693 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: enable_multiple, ever_enabled_multiple: 1,1 2021-03-09T09:00:58.693 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-03-09T09:00:58.693 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: legacy client fscid: -1 2021-03-09T09:00:58.693 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: 2021-03-09T09:00:58.694 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: No filesystems configured 2021-03-09T09:00:58.694 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.319+0000 7f9c7f9c3700 0 mon.c@-1(???).osd e18 crush map has features 3314932999778484224, adjusting msgr requires 2021-03-09T09:00:58.694 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.319+0000 7f9c7f9c3700 0 mon.c@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:00:58.694 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.319+0000 7f9c7f9c3700 0 mon.c@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:00:58.694 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.319+0000 7f9c7f9c3700 0 mon.c@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:00:58.695 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: audit 2021-03-09T08:59:48.920210+0000 mon.a (mon.0) 269 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T09:00:58.695 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: cluster 2021-03-09T08:59:48.920524+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-09T09:00:58.695 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: audit 2021-03-09T08:59:48.921424+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:00:58.695 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: audit 2021-03-09T08:59:48.930506+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:00:58.696 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.319+0000 7f9c7f9c3700 1 mon.c@-1(???).paxosservice(auth 1..11) refresh upgraded, format 0 -> 3 2021-03-09T09:00:58.696 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.323+0000 7f9c7f9c3700 20 expand_channel_meta expand map: {default=false} 2021-03-09T09:00:58.696 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.323+0000 7f9c7f9c3700 20 expand_channel_meta from 'false' to 'false' 2021-03-09T09:00:58.696 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.323+0000 7f9c7f9c3700 20 expand_channel_meta expanded map: {default=false} 2021-03-09T09:00:58.696 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.323+0000 7f9c7f9c3700 20 expand_channel_meta expand map: {default=info} 2021-03-09T09:00:58.697 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.323+0000 7f9c7f9c3700 20 expand_channel_meta from 'info' to 'info' 2021-03-09T09:00:58.697 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.323+0000 7f9c7f9c3700 20 expand_channel_meta expanded map: {default=info} 2021-03-09T09:00:58.697 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.323+0000 7f9c7f9c3700 20 expand_channel_meta expand map: {default=daemon} 2021-03-09T09:00:58.697 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.323+0000 7f9c7f9c3700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-03-09T09:00:58.697 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.323+0000 7f9c7f9c3700 20 expand_channel_meta expanded map: {default=daemon} 2021-03-09T09:00:58.698 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.323+0000 7f9c7f9c3700 20 expand_channel_meta expand map: {default=debug} 2021-03-09T09:00:58.698 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.323+0000 7f9c7f9c3700 20 expand_channel_meta from 'debug' to 'debug' 2021-03-09T09:00:58.698 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:00:58 smithi005 bash[25853]: debug 2021-03-09T09:00:58.323+0000 7f9c7f9c3700 20 expand_channel_meta expanded map: {default=debug} 2021-03-09T09:00:59.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:00:59 smithi146 bash[13136]: debug 2021-03-09T09:00:59.358+0000 7fa560337700 -1 mon.b@2(electing) e3 get_health_metrics reporting 59 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:01:04.365 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f9c6f9d5700 time 2021-03-09T09:01:04.037785+0000 2021-03-09T09:01:04.365 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:04.365 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:04.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f9c7ed5b1d8] 2021-03-09T09:01:04.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f9c7ed5b3f2] 2021-03-09T09:01:04.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x560d28f1997c] 2021-03-09T09:01:04.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x560d28f58ced] 2021-03-09T09:01:04.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x560d28f80b5a] 2021-03-09T09:01:04.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x560d28f8c054] 2021-03-09T09:01:04.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x560d28f902c3] 2021-03-09T09:01:04.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x560d28f11c2d] 2021-03-09T09:01:04.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x560d28e660b4] 2021-03-09T09:01:04.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 10: (C_MonOp::finish(int)+0x49) [0x560d28e05069] 2021-03-09T09:01:04.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 11: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x560d28e2dda5] 2021-03-09T09:01:04.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 13: (Paxos::finish_round()+0x27b) [0x560d28f0953b] 2021-03-09T09:01:04.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 14: (Paxos::commit_finish()+0x8c0) [0x560d28f0bf50] 2021-03-09T09:01:04.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 15: (C_Committed::finish(int)+0x45) [0x560d28f0fb85] 2021-03-09T09:01:04.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 16: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x560d28f0f8a8] 2021-03-09T09:01:04.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 18: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f9c7edfaaa5] 2021-03-09T09:01:04.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 20: (Thread::_entry_func(void*)+0xd) [0x7f9c7ee4241d] 2021-03-09T09:01:04.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f9c7c84114a] 2021-03-09T09:01:04.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 22: clone() 2021-03-09T09:01:04.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: *** Caught signal (Aborted) ** 2021-03-09T09:01:04.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: in thread 7f9c6f9d5700 thread_name:fn_monstore 2021-03-09T09:01:04.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: debug 2021-03-09T09:01:04.039+0000 7f9c6f9d5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f9c6f9d5700 time 2021-03-09T09:01:04.037785+0000 2021-03-09T09:01:04.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:04.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:04.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f9c7ed5b1d8] 2021-03-09T09:01:04.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f9c7ed5b3f2] 2021-03-09T09:01:04.374 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x560d28f1997c] 2021-03-09T09:01:04.374 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x560d28f58ced] 2021-03-09T09:01:04.374 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x560d28f80b5a] 2021-03-09T09:01:04.376 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x560d28f8c054] 2021-03-09T09:01:04.376 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x560d28f902c3] 2021-03-09T09:01:04.376 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x560d28f11c2d] 2021-03-09T09:01:04.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x560d28e660b4] 2021-03-09T09:01:04.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 10: (C_MonOp::finish(int)+0x49) [0x560d28e05069] 2021-03-09T09:01:04.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 11: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x560d28e2dda5] 2021-03-09T09:01:04.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 13: (Paxos::finish_round()+0x27b) [0x560d28f0953b] 2021-03-09T09:01:04.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 14: (Paxos::commit_finish()+0x8c0) [0x560d28f0bf50] 2021-03-09T09:01:04.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 15: (C_Committed::finish(int)+0x45) [0x560d28f0fb85] 2021-03-09T09:01:04.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 16: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x560d28f0f8a8] 2021-03-09T09:01:04.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 18: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f9c7edfaaa5] 2021-03-09T09:01:04.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 20: (Thread::_entry_func(void*)+0xd) [0x7f9c7ee4241d] 2021-03-09T09:01:04.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f9c7c84114a] 2021-03-09T09:01:04.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 22: clone() 2021-03-09T09:01:04.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:04.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9c7c84bb20] 2021-03-09T09:01:04.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 2: gsignal() 2021-03-09T09:01:04.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 3: abort() 2021-03-09T09:01:04.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f9c7ed5b229] 2021-03-09T09:01:04.382 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f9c7ed5b3f2] 2021-03-09T09:01:04.382 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x560d28f1997c] 2021-03-09T09:01:04.382 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x560d28f58ced] 2021-03-09T09:01:04.382 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x560d28f80b5a] 2021-03-09T09:01:04.383 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x560d28f8c054] 2021-03-09T09:01:04.383 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x560d28f902c3] 2021-03-09T09:01:04.383 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x560d28f11c2d] 2021-03-09T09:01:04.383 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x560d28e660b4] 2021-03-09T09:01:04.384 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 13: (C_MonOp::finish(int)+0x49) [0x560d28e05069] 2021-03-09T09:01:04.384 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 14: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.384 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x560d28e2dda5] 2021-03-09T09:01:04.384 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 16: (Paxos::finish_round()+0x27b) [0x560d28f0953b] 2021-03-09T09:01:04.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 17: (Paxos::commit_finish()+0x8c0) [0x560d28f0bf50] 2021-03-09T09:01:04.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 18: (C_Committed::finish(int)+0x45) [0x560d28f0fb85] 2021-03-09T09:01:04.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 19: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x560d28f0f8a8] 2021-03-09T09:01:04.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 21: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f9c7edfaaa5] 2021-03-09T09:01:04.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 23: (Thread::_entry_func(void*)+0xd) [0x7f9c7ee4241d] 2021-03-09T09:01:04.387 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f9c7c84114a] 2021-03-09T09:01:04.387 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 25: clone() 2021-03-09T09:01:04.387 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: debug 2021-03-09T09:01:04.047+0000 7f9c6f9d5700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:04.387 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: in thread 7f9c6f9d5700 thread_name:fn_monstore 2021-03-09T09:01:04.388 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:04.388 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9c7c84bb20] 2021-03-09T09:01:04.388 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 2: gsignal() 2021-03-09T09:01:04.389 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 3: abort() 2021-03-09T09:01:04.389 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f9c7ed5b229] 2021-03-09T09:01:04.389 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f9c7ed5b3f2] 2021-03-09T09:01:04.389 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x560d28f1997c] 2021-03-09T09:01:04.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x560d28f58ced] 2021-03-09T09:01:04.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x560d28f80b5a] 2021-03-09T09:01:04.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x560d28f8c054] 2021-03-09T09:01:04.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x560d28f902c3] 2021-03-09T09:01:04.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x560d28f11c2d] 2021-03-09T09:01:04.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x560d28e660b4] 2021-03-09T09:01:04.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 13: (C_MonOp::finish(int)+0x49) [0x560d28e05069] 2021-03-09T09:01:04.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 14: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x560d28e2dda5] 2021-03-09T09:01:04.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 16: (Paxos::finish_round()+0x27b) [0x560d28f0953b] 2021-03-09T09:01:04.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 17: (Paxos::commit_finish()+0x8c0) [0x560d28f0bf50] 2021-03-09T09:01:04.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 18: (C_Committed::finish(int)+0x45) [0x560d28f0fb85] 2021-03-09T09:01:04.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 19: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x560d28f0f8a8] 2021-03-09T09:01:04.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 21: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.394 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f9c7edfaaa5] 2021-03-09T09:01:04.394 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 23: (Thread::_entry_func(void*)+0xd) [0x7f9c7ee4241d] 2021-03-09T09:01:04.394 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f9c7c84114a] 2021-03-09T09:01:04.395 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 25: clone() 2021-03-09T09:01:04.395 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:04.395 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: debug -1> 2021-03-09T09:01:04.039+0000 7f9c6f9d5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f9c6f9d5700 time 2021-03-09T09:01:04.037785+0000 2021-03-09T09:01:04.395 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:04.396 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:04.396 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f9c7ed5b1d8] 2021-03-09T09:01:04.396 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f9c7ed5b3f2] 2021-03-09T09:01:04.396 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x560d28f1997c] 2021-03-09T09:01:04.397 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x560d28f58ced] 2021-03-09T09:01:04.397 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x560d28f80b5a] 2021-03-09T09:01:04.397 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x560d28f8c054] 2021-03-09T09:01:04.397 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x560d28f902c3] 2021-03-09T09:01:04.398 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x560d28f11c2d] 2021-03-09T09:01:04.398 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x560d28e660b4] 2021-03-09T09:01:04.398 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 10: (C_MonOp::finish(int)+0x49) [0x560d28e05069] 2021-03-09T09:01:04.398 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 11: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.398 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x560d28e2dda5] 2021-03-09T09:01:04.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 13: (Paxos::finish_round()+0x27b) [0x560d28f0953b] 2021-03-09T09:01:04.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 14: (Paxos::commit_finish()+0x8c0) [0x560d28f0bf50] 2021-03-09T09:01:04.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 15: (C_Committed::finish(int)+0x45) [0x560d28f0fb85] 2021-03-09T09:01:04.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 16: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.400 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x560d28f0f8a8] 2021-03-09T09:01:04.400 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 18: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.400 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f9c7edfaaa5] 2021-03-09T09:01:04.401 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 20: (Thread::_entry_func(void*)+0xd) [0x7f9c7ee4241d] 2021-03-09T09:01:04.401 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f9c7c84114a] 2021-03-09T09:01:04.401 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 22: clone() 2021-03-09T09:01:04.401 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: debug 0> 2021-03-09T09:01:04.047+0000 7f9c6f9d5700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:04.401 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: in thread 7f9c6f9d5700 thread_name:fn_monstore 2021-03-09T09:01:04.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:04.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9c7c84bb20] 2021-03-09T09:01:04.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 2: gsignal() 2021-03-09T09:01:04.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 3: abort() 2021-03-09T09:01:04.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f9c7ed5b229] 2021-03-09T09:01:04.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f9c7ed5b3f2] 2021-03-09T09:01:04.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x560d28f1997c] 2021-03-09T09:01:04.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x560d28f58ced] 2021-03-09T09:01:04.404 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x560d28f80b5a] 2021-03-09T09:01:04.404 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x560d28f8c054] 2021-03-09T09:01:04.404 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x560d28f902c3] 2021-03-09T09:01:04.405 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x560d28f11c2d] 2021-03-09T09:01:04.405 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x560d28e660b4] 2021-03-09T09:01:04.405 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 13: (C_MonOp::finish(int)+0x49) [0x560d28e05069] 2021-03-09T09:01:04.406 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 14: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.406 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x560d28e2dda5] 2021-03-09T09:01:04.406 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 16: (Paxos::finish_round()+0x27b) [0x560d28f0953b] 2021-03-09T09:01:04.407 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 17: (Paxos::commit_finish()+0x8c0) [0x560d28f0bf50] 2021-03-09T09:01:04.407 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 18: (C_Committed::finish(int)+0x45) [0x560d28f0fb85] 2021-03-09T09:01:04.407 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 19: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.407 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x560d28f0f8a8] 2021-03-09T09:01:04.408 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 21: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.408 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f9c7edfaaa5] 2021-03-09T09:01:04.408 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 23: (Thread::_entry_func(void*)+0xd) [0x7f9c7ee4241d] 2021-03-09T09:01:04.408 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f9c7c84114a] 2021-03-09T09:01:04.409 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 25: clone() 2021-03-09T09:01:04.409 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:04.409 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: debug -1> 2021-03-09T09:01:04.039+0000 7f9c6f9d5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f9c6f9d5700 time 2021-03-09T09:01:04.037785+0000 2021-03-09T09:01:04.409 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:04.410 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:04.410 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f9c7ed5b1d8] 2021-03-09T09:01:04.410 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f9c7ed5b3f2] 2021-03-09T09:01:04.410 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x560d28f1997c] 2021-03-09T09:01:04.411 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x560d28f58ced] 2021-03-09T09:01:04.411 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x560d28f80b5a] 2021-03-09T09:01:04.411 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x560d28f8c054] 2021-03-09T09:01:04.412 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x560d28f902c3] 2021-03-09T09:01:04.412 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x560d28f11c2d] 2021-03-09T09:01:04.412 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x560d28e660b4] 2021-03-09T09:01:04.412 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 10: (C_MonOp::finish(int)+0x49) [0x560d28e05069] 2021-03-09T09:01:04.413 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 11: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.413 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x560d28e2dda5] 2021-03-09T09:01:04.413 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 13: (Paxos::finish_round()+0x27b) [0x560d28f0953b] 2021-03-09T09:01:04.413 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 14: (Paxos::commit_finish()+0x8c0) [0x560d28f0bf50] 2021-03-09T09:01:04.414 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 15: (C_Committed::finish(int)+0x45) [0x560d28f0fb85] 2021-03-09T09:01:04.414 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 16: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.414 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x560d28f0f8a8] 2021-03-09T09:01:04.414 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 18: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.415 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f9c7edfaaa5] 2021-03-09T09:01:04.415 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 20: (Thread::_entry_func(void*)+0xd) [0x7f9c7ee4241d] 2021-03-09T09:01:04.415 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f9c7c84114a] 2021-03-09T09:01:04.415 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 22: clone() 2021-03-09T09:01:04.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: debug 0> 2021-03-09T09:01:04.047+0000 7f9c6f9d5700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:04.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: in thread 7f9c6f9d5700 thread_name:fn_monstore 2021-03-09T09:01:04.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:04.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9c7c84bb20] 2021-03-09T09:01:04.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 2: gsignal() 2021-03-09T09:01:04.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 3: abort() 2021-03-09T09:01:04.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f9c7ed5b229] 2021-03-09T09:01:04.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f9c7ed5b3f2] 2021-03-09T09:01:04.418 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x560d28f1997c] 2021-03-09T09:01:04.418 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x560d28f58ced] 2021-03-09T09:01:04.418 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x560d28f80b5a] 2021-03-09T09:01:04.419 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x560d28f8c054] 2021-03-09T09:01:04.419 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x560d28f902c3] 2021-03-09T09:01:04.419 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x560d28f11c2d] 2021-03-09T09:01:04.419 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x560d28e660b4] 2021-03-09T09:01:04.419 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 13: (C_MonOp::finish(int)+0x49) [0x560d28e05069] 2021-03-09T09:01:04.420 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 14: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.420 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x560d28e2dda5] 2021-03-09T09:01:04.420 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 16: (Paxos::finish_round()+0x27b) [0x560d28f0953b] 2021-03-09T09:01:04.420 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 17: (Paxos::commit_finish()+0x8c0) [0x560d28f0bf50] 2021-03-09T09:01:04.421 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 18: (C_Committed::finish(int)+0x45) [0x560d28f0fb85] 2021-03-09T09:01:04.421 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 19: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.421 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x560d28f0f8a8] 2021-03-09T09:01:04.421 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 21: (Context::complete(int)+0xd) [0x560d28e0249d] 2021-03-09T09:01:04.421 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f9c7edfaaa5] 2021-03-09T09:01:04.422 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 23: (Thread::_entry_func(void*)+0xd) [0x7f9c7ee4241d] 2021-03-09T09:01:04.422 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f9c7c84114a] 2021-03-09T09:01:04.422 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: 25: clone() 2021-03-09T09:01:04.422 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 bash[25853]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:04.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:01:04 smithi146 bash[13136]: debug 2021-03-09T09:01:04.358+0000 7fa560337700 -1 mon.b@2(peon) e3 get_health_metrics reporting 69 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:01:05.364 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Main process exited, code=exited, status=134/n/a 2021-03-09T09:01:05.364 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:04 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Failed with result 'exit-code'. 2021-03-09T09:01:07.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:06 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Scheduled restart job, restart counter is at 3. 2021-03-09T09:01:07.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:06 smithi005 systemd[1]: Stopped Ceph mon.a for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:01:07.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:06 smithi005 systemd[1]: Started Ceph mon.a for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:01:08.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.859+0000 7fbebaed4700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-09T09:01:08.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.859+0000 7fbebaed4700 0 ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev), process ceph-mon, pid 6 2021-03-09T09:01:08.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.859+0000 7fbebaed4700 0 pidfile_write: ignore empty --pid-file 2021-03-09T09:01:08.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 0 load: jerasure load: lrc load: isa 2021-03-09T09:01:08.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: RocksDB version: 6.8.1 2021-03-09T09:01:08.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-09T09:01:08.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Compile date Mar 9 2021 2021-03-09T09:01:08.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: DB SUMMARY 2021-03-09T09:01:08.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: CURRENT file: CURRENT 2021-03-09T09:01:08.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-09T09:01:08.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: MANIFEST file: MANIFEST-000014 size: 446 Bytes 2021-03-09T09:01:08.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000018.sst 2021-03-09T09:01:08.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000015.log size: 18952 ; 2021-03-09T09:01:08.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.error_if_exists: 0 2021-03-09T09:01:08.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.create_if_missing: 0 2021-03-09T09:01:08.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.paranoid_checks: 1 2021-03-09T09:01:08.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.env: 0x56160db121c0 2021-03-09T09:01:08.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.fs: Posix File System 2021-03-09T09:01:08.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.info_log: 0x561617c6f120 2021-03-09T09:01:08.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-09T09:01:08.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.statistics: (nil) 2021-03-09T09:01:08.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.use_fsync: 0 2021-03-09T09:01:08.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_log_file_size: 0 2021-03-09T09:01:08.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-09T09:01:08.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-09T09:01:08.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-09T09:01:08.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-09T09:01:08.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.allow_fallocate: 1 2021-03-09T09:01:08.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-09T09:01:08.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-09T09:01:08.123 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.use_direct_reads: 0 2021-03-09T09:01:08.123 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-09T09:01:08.123 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-09T09:01:08.123 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.db_log_dir: 2021-03-09T09:01:08.123 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2021-03-09T09:01:08.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-09T09:01:08.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_subcompactions: 1 2021-03-09T09:01:08.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_background_flushes: -1 2021-03-09T09:01:08.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-09T09:01:08.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-09T09:01:08.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-09T09:01:08.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-09T09:01:08.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-09T09:01:08.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-09T09:01:08.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-09T09:01:08.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.write_buffer_manager: 0x561617cc3cb0 2021-03-09T09:01:08.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-09T09:01:08.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-09T09:01:08.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-09T09:01:08.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-09T09:01:08.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-09T09:01:08.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-09T09:01:08.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-09T09:01:08.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-09T09:01:08.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-09T09:01:08.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.unordered_write: 0 2021-03-09T09:01:08.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-09T09:01:08.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-09T09:01:08.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-09T09:01:08.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-09T09:01:08.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.row_cache: None 2021-03-09T09:01:08.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.wal_filter: None 2021-03-09T09:01:08.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-09T09:01:08.130 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-09T09:01:08.130 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.preserve_deletes: 0 2021-03-09T09:01:08.130 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.two_write_queues: 0 2021-03-09T09:01:08.130 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-09T09:01:08.130 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.atomic_flush: 0 2021-03-09T09:01:08.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-09T09:01:08.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-09T09:01:08.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-09T09:01:08.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.log_readahead_size: 0 2021-03-09T09:01:08.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-09T09:01:08.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_background_jobs: 2 2021-03-09T09:01:08.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_background_compactions: -1 2021-03-09T09:01:08.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-09T09:01:08.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-09T09:01:08.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-09T09:01:08.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-09T09:01:08.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-09T09:01:08.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-09T09:01:08.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-09T09:01:08.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-09T09:01:08.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_open_files: -1 2021-03-09T09:01:08.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-09T09:01:08.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-09T09:01:08.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-09T09:01:08.135 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-09T09:01:08.135 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Compression algorithms supported: 2021-03-09T09:01:08.135 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-09T09:01:08.135 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: kZSTD supported: 0 2021-03-09T09:01:08.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: kXpressCompression supported: 0 2021-03-09T09:01:08.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-09T09:01:08.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: kLZ4Compression supported: 1 2021-03-09T09:01:08.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: kBZip2Compression supported: 0 2021-03-09T09:01:08.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: kZlibCompression supported: 1 2021-03-09T09:01:08.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: kSnappyCompression supported: 1 2021-03-09T09:01:08.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-09T09:01:08.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000014 2021-03-09T09:01:08.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-09T09:01:08.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-09T09:01:08.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.merge_operator: 2021-03-09T09:01:08.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compaction_filter: None 2021-03-09T09:01:08.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-09T09:01:08.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-09T09:01:08.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-09T09:01:08.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56160fa12230) 2021-03-09T09:01:08.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: cache_index_and_filter_blocks: 1 2021-03-09T09:01:08.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-09T09:01:08.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-09T09:01:08.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: pin_top_level_index_and_filter: 1 2021-03-09T09:01:08.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: index_type: 0 2021-03-09T09:01:08.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: data_block_index_type: 0 2021-03-09T09:01:08.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: index_shortening: 1 2021-03-09T09:01:08.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: data_block_hash_table_util_ratio: 0.750000 2021-03-09T09:01:08.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: hash_index_allow_collision: 1 2021-03-09T09:01:08.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: checksum: 1 2021-03-09T09:01:08.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: no_block_cache: 0 2021-03-09T09:01:08.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: block_cache: 0x561617c4a510 2021-03-09T09:01:08.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: block_cache_name: BinnedLRUCache 2021-03-09T09:01:08.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: block_cache_options: 2021-03-09T09:01:08.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: capacity : 536870912 2021-03-09T09:01:08.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: num_shard_bits : 4 2021-03-09T09:01:08.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: strict_capacity_limit : 0 2021-03-09T09:01:08.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: high_pri_pool_ratio: 0.000 2021-03-09T09:01:08.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: block_cache_compressed: (nil) 2021-03-09T09:01:08.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: persistent_cache: (nil) 2021-03-09T09:01:08.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: block_size: 4096 2021-03-09T09:01:08.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: block_size_deviation: 10 2021-03-09T09:01:08.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: block_restart_interval: 16 2021-03-09T09:01:08.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: index_block_restart_interval: 1 2021-03-09T09:01:08.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: metadata_block_size: 4096 2021-03-09T09:01:08.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: partition_filters: 0 2021-03-09T09:01:08.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: use_delta_encoding: 1 2021-03-09T09:01:08.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-09T09:01:08.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: whole_key_filtering: 1 2021-03-09T09:01:08.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: verify_compression: 0 2021-03-09T09:01:08.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: read_amp_bytes_per_bit: 0 2021-03-09T09:01:08.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: format_version: 2 2021-03-09T09:01:08.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: enable_index_compression: 1 2021-03-09T09:01:08.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: block_align: 0 2021-03-09T09:01:08.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-09T09:01:08.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-09T09:01:08.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compression: NoCompression 2021-03-09T09:01:08.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-09T09:01:08.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-09T09:01:08.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-09T09:01:08.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.num_levels: 7 2021-03-09T09:01:08.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-09T09:01:08.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-09T09:01:08.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-09T09:01:08.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-09T09:01:08.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-09T09:01:08.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-09T09:01:08.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-09T09:01:08.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:01:08.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-09T09:01:08.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-09T09:01:08.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-09T09:01:08.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-09T09:01:08.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-09T09:01:08.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:01:08.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-09T09:01:08.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-09T09:01:08.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-09T09:01:08.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-09T09:01:08.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-09T09:01:08.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-09T09:01:08.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-09T09:01:08.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-09T09:01:08.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-09T09:01:08.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-09T09:01:08.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-09T09:01:08.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-09T09:01:08.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-09T09:01:08.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-09T09:01:08.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-09T09:01:08.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-09T09:01:08.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-09T09:01:08.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-09T09:01:08.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-09T09:01:08.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-09T09:01:08.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-09T09:01:08.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-09T09:01:08.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-09T09:01:08.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-09T09:01:08.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-09T09:01:08.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-09T09:01:08.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-09T09:01:08.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-09T09:01:08.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-09T09:01:08.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-09T09:01:08.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-09T09:01:08.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-09T09:01:08.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-09T09:01:08.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.table_properties_collectors: 2021-03-09T09:01:08.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.inplace_update_support: 0 2021-03-09T09:01:08.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-09T09:01:08.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-09T09:01:08.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-09T09:01:08.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-09T09:01:08.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.bloom_locality: 0 2021-03-09T09:01:08.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.max_successive_merges: 0 2021-03-09T09:01:08.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-09T09:01:08.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-09T09:01:08.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-09T09:01:08.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-09T09:01:08.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.ttl: 2592000 2021-03-09T09:01:08.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-09T09:01:08.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: [version_set.cc:3291] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2021-03-09T09:01:08.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000014 succeeded,manifest_file_number is 14, next_file_number is 20, last_sequence is 3260, log_number is 13,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-09T09:01:08.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 13 2021-03-09T09:01:08.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280467870165, "job": 1, "event": "recovery_started", "log_files": [15]} 2021-03-09T09:01:08.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #15 mode 2 2021-03-09T09:01:08.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-09T09:01:08.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280467871270, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 20, "file_size": 15014, "table_properties": {"data_size": 14049, "index_size": 90, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 359, "raw_average_key_size": 23, "raw_value_size": 13688, "raw_average_value_size": 912, "num_data_blocks": 3, "num_entries": 15, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1615280467, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-09T09:01:08.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: [version_set.cc:3826] Creating manifest 21 2021-03-09T09:01:08.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.867+0000 7fbebaed4700 4 rocksdb: [version_set.cc:3291] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2021-03-09T09:01:08.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.871+0000 7fbebaed4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280467872585, "job": 1, "event": "recovery_finished"} 2021-03-09T09:01:08.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.871+0000 7fbebaed4700 4 rocksdb: DB pointer 0x561617d0f800 2021-03-09T09:01:08.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.871+0000 7fbea9ee4700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-09T09:01:08.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.871+0000 7fbea9ee4700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-09T09:01:08.163 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: ** DB Stats ** 2021-03-09T09:01:08.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:01:08.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-09T09:01:08.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-09T09:01:08.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:01:08.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-09T09:01:08.165 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-09T09:01:08.165 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:01:08.165 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: ** Compaction Stats [default] ** 2021-03-09T09:01:08.165 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:08.165 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:08.166 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: L0 1/0 14.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 16.5 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:08.166 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: L6 1/0 7.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:01:08.166 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Sum 2/0 7.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 16.5 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:08.166 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 16.5 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:08.166 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: ** Compaction Stats [default] ** 2021-03-09T09:01:08.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:08.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:08.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 16.5 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:08.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:01:08.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:08.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:08.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:01:08.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:01:08.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:01:08.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Cumulative compaction: 0.00 GB write, 2.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:08.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Interval compaction: 0.00 GB write, 2.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:08.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:01:08.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:01:08.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: ** Compaction Stats [default] ** 2021-03-09T09:01:08.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:08.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:08.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: L0 1/0 14.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 16.5 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:08.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: L6 1/0 7.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:01:08.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Sum 2/0 7.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 16.5 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:08.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:01:08.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: ** Compaction Stats [default] ** 2021-03-09T09:01:08.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:08.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:08.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 16.5 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:08.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:01:08.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:08.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:08.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:01:08.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:01:08.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:01:08.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Cumulative compaction: 0.00 GB write, 2.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:08.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:08.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:01:08.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:01:08.174 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.875+0000 7fbebaed4700 0 starting mon.a rank 0 at public addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] at bind addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:01:08.174 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 1 mon.a@-1(???) e3 preinit fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:01:08.174 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 0 mon.a@-1(???).mds e1 new map 2021-03-09T09:01:08.174 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 0 mon.a@-1(???).mds e1 print_map 2021-03-09T09:01:08.174 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: e1 2021-03-09T09:01:08.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: enable_multiple, ever_enabled_multiple: 1,1 2021-03-09T09:01:08.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-03-09T09:01:08.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: legacy client fscid: -1 2021-03-09T09:01:08.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: 2021-03-09T09:01:08.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: No filesystems configured 2021-03-09T09:01:08.176 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 0 mon.a@-1(???).osd e18 crush map has features 3314932999778484224, adjusting msgr requires 2021-03-09T09:01:08.176 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 0 mon.a@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:01:08.176 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 0 mon.a@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:01:08.176 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 0 mon.a@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:01:08.177 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: audit 2021-03-09T08:59:48.920210+0000 mon.a (mon.0) 269 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T09:01:08.177 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: cluster 2021-03-09T08:59:48.920524+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-09T09:01:08.177 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: audit 2021-03-09T08:59:48.921424+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:01:08.177 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: audit 2021-03-09T08:59:48.930506+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:01:08.177 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 1 mon.a@-1(???).paxosservice(auth 1..11) refresh upgraded, format 0 -> 3 2021-03-09T09:01:08.178 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 20 expand_channel_meta expand map: {default=false} 2021-03-09T09:01:08.178 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 20 expand_channel_meta from 'false' to 'false' 2021-03-09T09:01:08.178 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 20 expand_channel_meta expanded map: {default=false} 2021-03-09T09:01:08.178 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 20 expand_channel_meta expand map: {default=info} 2021-03-09T09:01:08.178 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 20 expand_channel_meta from 'info' to 'info' 2021-03-09T09:01:08.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 20 expand_channel_meta expanded map: {default=info} 2021-03-09T09:01:08.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 20 expand_channel_meta expand map: {default=daemon} 2021-03-09T09:01:08.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-03-09T09:01:08.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 20 expand_channel_meta expanded map: {default=daemon} 2021-03-09T09:01:08.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 20 expand_channel_meta expand map: {default=debug} 2021-03-09T09:01:08.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 20 expand_channel_meta from 'debug' to 'debug' 2021-03-09T09:01:08.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:07 smithi005 bash[25980]: debug 2021-03-09T09:01:07.879+0000 7fbebaed4700 20 expand_channel_meta expanded map: {default=debug} 2021-03-09T09:01:09.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:01:09 smithi146 bash[13136]: debug 2021-03-09T09:01:09.362+0000 7fa560337700 -1 mon.b@2(peon) e3 get_health_metrics reporting 79 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:01:14.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:01:14 smithi146 bash[13136]: debug 2021-03-09T09:01:14.358+0000 7fa560337700 -1 mon.b@2(electing) e3 get_health_metrics reporting 90 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:01:15.613 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:15 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Scheduled restart job, restart counter is at 4. 2021-03-09T09:01:15.614 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:15 smithi005 systemd[1]: Stopped Ceph mon.c for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:01:15.614 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:15 smithi005 systemd[1]: Started Ceph mon.c for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:01:16.484 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fbeaaee6700 time 2021-03-09T09:01:16.419103+0000 2021-03-09T09:01:16.484 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:16.485 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:16.485 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fbeba26c1d8] 2021-03-09T09:01:16.485 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fbeba26c3f2] 2021-03-09T09:01:16.485 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x56160d37497c] 2021-03-09T09:01:16.486 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x56160d3b3ced] 2021-03-09T09:01:16.486 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x56160d3dbb5a] 2021-03-09T09:01:16.486 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x56160d3e7054] 2021-03-09T09:01:16.486 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x56160d3eb2c3] 2021-03-09T09:01:16.487 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x56160d36cc2d] 2021-03-09T09:01:16.487 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x56160d2c10b4] 2021-03-09T09:01:16.487 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 10: (C_MonOp::finish(int)+0x49) [0x56160d260069] 2021-03-09T09:01:16.487 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 11: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.487 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x56160d288da5] 2021-03-09T09:01:16.488 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 13: (Paxos::finish_round()+0x27b) [0x56160d36453b] 2021-03-09T09:01:16.488 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.375+0000 7fe468786700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-09T09:01:16.489 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.375+0000 7fe468786700 0 ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev), process ceph-mon, pid 6 2021-03-09T09:01:16.489 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.375+0000 7fe468786700 0 pidfile_write: ignore empty --pid-file 2021-03-09T09:01:16.489 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 0 load: jerasure load: lrc load: isa 2021-03-09T09:01:16.490 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: RocksDB version: 6.8.1 2021-03-09T09:01:16.490 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-09T09:01:16.490 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Compile date Mar 9 2021 2021-03-09T09:01:16.490 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: DB SUMMARY 2021-03-09T09:01:16.491 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: CURRENT file: CURRENT 2021-03-09T09:01:16.491 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-09T09:01:16.491 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: MANIFEST file: MANIFEST-000014 size: 451 Bytes 2021-03-09T09:01:16.491 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000018.sst 2021-03-09T09:01:16.492 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000015.log size: 18634 ; 2021-03-09T09:01:16.492 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.error_if_exists: 0 2021-03-09T09:01:16.492 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.create_if_missing: 0 2021-03-09T09:01:16.492 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.paranoid_checks: 1 2021-03-09T09:01:16.492 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.env: 0x55f35d7d51c0 2021-03-09T09:01:16.493 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.fs: Posix File System 2021-03-09T09:01:16.493 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.info_log: 0x55f366665120 2021-03-09T09:01:16.493 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-09T09:01:16.493 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.statistics: (nil) 2021-03-09T09:01:16.493 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.use_fsync: 0 2021-03-09T09:01:16.494 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_log_file_size: 0 2021-03-09T09:01:16.494 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-09T09:01:16.494 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-09T09:01:16.494 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-09T09:01:16.495 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-09T09:01:16.495 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 14: (Paxos::commit_finish()+0x8c0) [0x56160d366f50] 2021-03-09T09:01:16.495 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 15: (C_Committed::finish(int)+0x45) [0x56160d36ab85] 2021-03-09T09:01:16.496 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 16: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.496 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x56160d36a8a8] 2021-03-09T09:01:16.496 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 18: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.496 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7fbeba30baa5] 2021-03-09T09:01:16.497 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 20: (Thread::_entry_func(void*)+0xd) [0x7fbeba35341d] 2021-03-09T09:01:16.497 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 21: /lib64/libpthread.so.0(+0x814a) [0x7fbeb7d5214a] 2021-03-09T09:01:16.497 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 22: clone() 2021-03-09T09:01:16.497 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: *** Caught signal (Aborted) ** 2021-03-09T09:01:16.497 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: in thread 7fbeaaee6700 thread_name:fn_monstore 2021-03-09T09:01:16.498 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: debug 2021-03-09T09:01:16.419+0000 7fbeaaee6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fbeaaee6700 time 2021-03-09T09:01:16.419103+0000 2021-03-09T09:01:16.498 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:16.498 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:16.498 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fbeba26c1d8] 2021-03-09T09:01:16.499 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fbeba26c3f2] 2021-03-09T09:01:16.499 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x56160d37497c] 2021-03-09T09:01:16.500 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x56160d3b3ced] 2021-03-09T09:01:16.500 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x56160d3dbb5a] 2021-03-09T09:01:16.500 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.allow_fallocate: 1 2021-03-09T09:01:16.501 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-09T09:01:16.501 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-09T09:01:16.501 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.use_direct_reads: 0 2021-03-09T09:01:16.501 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-09T09:01:16.501 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-09T09:01:16.502 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.db_log_dir: 2021-03-09T09:01:16.502 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2021-03-09T09:01:16.502 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-09T09:01:16.502 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_subcompactions: 1 2021-03-09T09:01:16.503 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_background_flushes: -1 2021-03-09T09:01:16.503 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-09T09:01:16.503 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-09T09:01:16.503 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-09T09:01:16.503 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-09T09:01:16.504 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-09T09:01:16.504 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-09T09:01:16.504 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-09T09:01:16.504 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.write_buffer_manager: 0x55f3666b9ce0 2021-03-09T09:01:16.504 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-09T09:01:16.505 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-09T09:01:16.505 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-09T09:01:16.505 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-09T09:01:16.505 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-09T09:01:16.505 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-09T09:01:16.506 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-09T09:01:16.506 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-09T09:01:16.506 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x56160d3e7054] 2021-03-09T09:01:16.507 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x56160d3eb2c3] 2021-03-09T09:01:16.507 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x56160d36cc2d] 2021-03-09T09:01:16.507 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x56160d2c10b4] 2021-03-09T09:01:16.507 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 10: (C_MonOp::finish(int)+0x49) [0x56160d260069] 2021-03-09T09:01:16.508 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 11: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.508 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x56160d288da5] 2021-03-09T09:01:16.508 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 13: (Paxos::finish_round()+0x27b) [0x56160d36453b] 2021-03-09T09:01:16.508 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 14: (Paxos::commit_finish()+0x8c0) [0x56160d366f50] 2021-03-09T09:01:16.508 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 15: (C_Committed::finish(int)+0x45) [0x56160d36ab85] 2021-03-09T09:01:16.509 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 16: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.509 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x56160d36a8a8] 2021-03-09T09:01:16.509 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 18: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.509 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7fbeba30baa5] 2021-03-09T09:01:16.509 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 20: (Thread::_entry_func(void*)+0xd) [0x7fbeba35341d] 2021-03-09T09:01:16.510 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 21: /lib64/libpthread.so.0(+0x814a) [0x7fbeb7d5214a] 2021-03-09T09:01:16.510 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 22: clone() 2021-03-09T09:01:16.510 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:16.510 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fbeb7d5cb20] 2021-03-09T09:01:16.511 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 2: gsignal() 2021-03-09T09:01:16.511 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 3: abort() 2021-03-09T09:01:16.511 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fbeba26c229] 2021-03-09T09:01:16.511 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fbeba26c3f2] 2021-03-09T09:01:16.511 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x56160d37497c] 2021-03-09T09:01:16.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x56160d3b3ced] 2021-03-09T09:01:16.512 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-09T09:01:16.512 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.unordered_write: 0 2021-03-09T09:01:16.513 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-09T09:01:16.513 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-09T09:01:16.513 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-09T09:01:16.513 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-09T09:01:16.513 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.row_cache: None 2021-03-09T09:01:16.514 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.wal_filter: None 2021-03-09T09:01:16.514 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-09T09:01:16.514 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-09T09:01:16.514 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.preserve_deletes: 0 2021-03-09T09:01:16.515 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.two_write_queues: 0 2021-03-09T09:01:16.515 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-09T09:01:16.515 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.atomic_flush: 0 2021-03-09T09:01:16.515 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-09T09:01:16.515 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-09T09:01:16.516 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-09T09:01:16.516 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.log_readahead_size: 0 2021-03-09T09:01:16.516 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-09T09:01:16.516 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_background_jobs: 2 2021-03-09T09:01:16.516 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_background_compactions: -1 2021-03-09T09:01:16.517 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-09T09:01:16.517 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-09T09:01:16.517 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-09T09:01:16.517 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-09T09:01:16.517 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-09T09:01:16.518 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-09T09:01:16.518 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x56160d3dbb5a] 2021-03-09T09:01:16.519 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x56160d3e7054] 2021-03-09T09:01:16.519 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x56160d3eb2c3] 2021-03-09T09:01:16.519 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x56160d36cc2d] 2021-03-09T09:01:16.519 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x56160d2c10b4] 2021-03-09T09:01:16.519 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 13: (C_MonOp::finish(int)+0x49) [0x56160d260069] 2021-03-09T09:01:16.520 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 14: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.520 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x56160d288da5] 2021-03-09T09:01:16.520 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 16: (Paxos::finish_round()+0x27b) [0x56160d36453b] 2021-03-09T09:01:16.520 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 17: (Paxos::commit_finish()+0x8c0) [0x56160d366f50] 2021-03-09T09:01:16.520 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 18: (C_Committed::finish(int)+0x45) [0x56160d36ab85] 2021-03-09T09:01:16.521 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 19: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.521 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x56160d36a8a8] 2021-03-09T09:01:16.521 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 21: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.521 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7fbeba30baa5] 2021-03-09T09:01:16.521 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 23: (Thread::_entry_func(void*)+0xd) [0x7fbeba35341d] 2021-03-09T09:01:16.522 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 24: /lib64/libpthread.so.0(+0x814a) [0x7fbeb7d5214a] 2021-03-09T09:01:16.522 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 25: clone() 2021-03-09T09:01:16.522 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: debug 2021-03-09T09:01:16.419+0000 7fbeaaee6700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:16.522 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: in thread 7fbeaaee6700 thread_name:fn_monstore 2021-03-09T09:01:16.523 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:16.523 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fbeb7d5cb20] 2021-03-09T09:01:16.523 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 2: gsignal() 2021-03-09T09:01:16.523 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 3: abort() 2021-03-09T09:01:16.523 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fbeba26c229] 2021-03-09T09:01:16.524 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fbeba26c3f2] 2021-03-09T09:01:16.524 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x56160d37497c] 2021-03-09T09:01:16.524 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x56160d3b3ced] 2021-03-09T09:01:16.525 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-09T09:01:16.525 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-09T09:01:16.525 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_open_files: -1 2021-03-09T09:01:16.525 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-09T09:01:16.526 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-09T09:01:16.526 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-09T09:01:16.526 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-09T09:01:16.526 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Compression algorithms supported: 2021-03-09T09:01:16.526 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-09T09:01:16.527 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: kZSTD supported: 0 2021-03-09T09:01:16.527 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: kXpressCompression supported: 0 2021-03-09T09:01:16.527 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-09T09:01:16.527 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: kLZ4Compression supported: 1 2021-03-09T09:01:16.528 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: kBZip2Compression supported: 0 2021-03-09T09:01:16.528 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: kZlibCompression supported: 1 2021-03-09T09:01:16.528 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: kSnappyCompression supported: 1 2021-03-09T09:01:16.528 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-09T09:01:16.528 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000014 2021-03-09T09:01:16.529 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-09T09:01:16.529 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-09T09:01:16.529 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.merge_operator: 2021-03-09T09:01:16.529 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.compaction_filter: None 2021-03-09T09:01:16.529 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-09T09:01:16.530 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-09T09:01:16.530 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-09T09:01:16.530 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f35e408230) 2021-03-09T09:01:16.530 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: cache_index_and_filter_blocks: 1 2021-03-09T09:01:16.530 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-09T09:01:16.531 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-09T09:01:16.531 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: pin_top_level_index_and_filter: 1 2021-03-09T09:01:16.532 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x56160d3dbb5a] 2021-03-09T09:01:16.532 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x56160d3e7054] 2021-03-09T09:01:16.532 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x56160d3eb2c3] 2021-03-09T09:01:16.532 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x56160d36cc2d] 2021-03-09T09:01:16.533 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x56160d2c10b4] 2021-03-09T09:01:16.533 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 13: (C_MonOp::finish(int)+0x49) [0x56160d260069] 2021-03-09T09:01:16.533 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 14: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.533 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x56160d288da5] 2021-03-09T09:01:16.534 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 16: (Paxos::finish_round()+0x27b) [0x56160d36453b] 2021-03-09T09:01:16.534 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 17: (Paxos::commit_finish()+0x8c0) [0x56160d366f50] 2021-03-09T09:01:16.534 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 18: (C_Committed::finish(int)+0x45) [0x56160d36ab85] 2021-03-09T09:01:16.534 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 19: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.535 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x56160d36a8a8] 2021-03-09T09:01:16.535 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 21: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.535 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7fbeba30baa5] 2021-03-09T09:01:16.535 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 23: (Thread::_entry_func(void*)+0xd) [0x7fbeba35341d] 2021-03-09T09:01:16.535 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 24: /lib64/libpthread.so.0(+0x814a) [0x7fbeb7d5214a] 2021-03-09T09:01:16.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 25: clone() 2021-03-09T09:01:16.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:16.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: debug -1> 2021-03-09T09:01:16.419+0000 7fbeaaee6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fbeaaee6700 time 2021-03-09T09:01:16.419103+0000 2021-03-09T09:01:16.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:16.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:16.537 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fbeba26c1d8] 2021-03-09T09:01:16.537 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fbeba26c3f2] 2021-03-09T09:01:16.537 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x56160d37497c] 2021-03-09T09:01:16.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x56160d3b3ced] 2021-03-09T09:01:16.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x56160d3dbb5a] 2021-03-09T09:01:16.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x56160d3e7054] 2021-03-09T09:01:16.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x56160d3eb2c3] 2021-03-09T09:01:16.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x56160d36cc2d] 2021-03-09T09:01:16.539 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x56160d2c10b4] 2021-03-09T09:01:16.539 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 10: (C_MonOp::finish(int)+0x49) [0x56160d260069] 2021-03-09T09:01:16.539 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 11: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x56160d288da5] 2021-03-09T09:01:16.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 13: (Paxos::finish_round()+0x27b) [0x56160d36453b] 2021-03-09T09:01:16.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 14: (Paxos::commit_finish()+0x8c0) [0x56160d366f50] 2021-03-09T09:01:16.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 15: (C_Committed::finish(int)+0x45) [0x56160d36ab85] 2021-03-09T09:01:16.541 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 16: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.541 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x56160d36a8a8] 2021-03-09T09:01:16.541 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 18: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.541 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7fbeba30baa5] 2021-03-09T09:01:16.541 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 20: (Thread::_entry_func(void*)+0xd) [0x7fbeba35341d] 2021-03-09T09:01:16.542 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 21: /lib64/libpthread.so.0(+0x814a) [0x7fbeb7d5214a] 2021-03-09T09:01:16.542 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 22: clone() 2021-03-09T09:01:16.542 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: debug 0> 2021-03-09T09:01:16.419+0000 7fbeaaee6700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:16.542 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: in thread 7fbeaaee6700 thread_name:fn_monstore 2021-03-09T09:01:16.542 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:16.543 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fbeb7d5cb20] 2021-03-09T09:01:16.543 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 2: gsignal() 2021-03-09T09:01:16.543 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 3: abort() 2021-03-09T09:01:16.543 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fbeba26c229] 2021-03-09T09:01:16.544 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: index_type: 0 2021-03-09T09:01:16.544 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: data_block_index_type: 0 2021-03-09T09:01:16.544 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: index_shortening: 1 2021-03-09T09:01:16.544 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: data_block_hash_table_util_ratio: 0.750000 2021-03-09T09:01:16.545 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: hash_index_allow_collision: 1 2021-03-09T09:01:16.545 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: checksum: 1 2021-03-09T09:01:16.545 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: no_block_cache: 0 2021-03-09T09:01:16.545 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: block_cache: 0x55f366640510 2021-03-09T09:01:16.545 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: block_cache_name: BinnedLRUCache 2021-03-09T09:01:16.546 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: block_cache_options: 2021-03-09T09:01:16.546 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: capacity : 536870912 2021-03-09T09:01:16.546 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: num_shard_bits : 4 2021-03-09T09:01:16.546 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: strict_capacity_limit : 0 2021-03-09T09:01:16.546 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: high_pri_pool_ratio: 0.000 2021-03-09T09:01:16.547 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: block_cache_compressed: (nil) 2021-03-09T09:01:16.547 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: persistent_cache: (nil) 2021-03-09T09:01:16.547 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: block_size: 4096 2021-03-09T09:01:16.547 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: block_size_deviation: 10 2021-03-09T09:01:16.547 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: block_restart_interval: 16 2021-03-09T09:01:16.548 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: index_block_restart_interval: 1 2021-03-09T09:01:16.548 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: metadata_block_size: 4096 2021-03-09T09:01:16.548 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: partition_filters: 0 2021-03-09T09:01:16.548 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: use_delta_encoding: 1 2021-03-09T09:01:16.549 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-09T09:01:16.549 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: whole_key_filtering: 1 2021-03-09T09:01:16.549 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: verify_compression: 0 2021-03-09T09:01:16.549 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: read_amp_bytes_per_bit: 0 2021-03-09T09:01:16.549 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: format_version: 2 2021-03-09T09:01:16.550 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: enable_index_compression: 1 2021-03-09T09:01:16.550 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: block_align: 0 2021-03-09T09:01:16.550 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-09T09:01:16.550 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-09T09:01:16.550 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.compression: NoCompression 2021-03-09T09:01:16.551 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-09T09:01:16.551 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-09T09:01:16.551 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-09T09:01:16.551 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.num_levels: 7 2021-03-09T09:01:16.551 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-09T09:01:16.552 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-09T09:01:16.552 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-09T09:01:16.552 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-09T09:01:16.552 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-09T09:01:16.552 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-09T09:01:16.553 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-09T09:01:16.553 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fbeba26c3f2] 2021-03-09T09:01:16.553 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x56160d37497c] 2021-03-09T09:01:16.554 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x56160d3b3ced] 2021-03-09T09:01:16.554 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x56160d3dbb5a] 2021-03-09T09:01:16.554 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x56160d3e7054] 2021-03-09T09:01:16.554 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x56160d3eb2c3] 2021-03-09T09:01:16.555 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x56160d36cc2d] 2021-03-09T09:01:16.555 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x56160d2c10b4] 2021-03-09T09:01:16.555 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 13: (C_MonOp::finish(int)+0x49) [0x56160d260069] 2021-03-09T09:01:16.555 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 14: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.555 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x56160d288da5] 2021-03-09T09:01:16.556 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 16: (Paxos::finish_round()+0x27b) [0x56160d36453b] 2021-03-09T09:01:16.556 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 17: (Paxos::commit_finish()+0x8c0) [0x56160d366f50] 2021-03-09T09:01:16.556 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 18: (C_Committed::finish(int)+0x45) [0x56160d36ab85] 2021-03-09T09:01:16.556 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 19: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.556 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x56160d36a8a8] 2021-03-09T09:01:16.557 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 21: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.557 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7fbeba30baa5] 2021-03-09T09:01:16.557 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 23: (Thread::_entry_func(void*)+0xd) [0x7fbeba35341d] 2021-03-09T09:01:16.557 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 24: /lib64/libpthread.so.0(+0x814a) [0x7fbeb7d5214a] 2021-03-09T09:01:16.557 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 25: clone() 2021-03-09T09:01:16.558 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:16.558 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:01:16.558 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-09T09:01:16.559 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-09T09:01:16.559 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-09T09:01:16.559 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-09T09:01:16.559 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-09T09:01:16.560 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:01:16.560 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-09T09:01:16.560 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-09T09:01:16.560 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-09T09:01:16.560 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-09T09:01:16.561 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-09T09:01:16.561 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-09T09:01:16.561 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-09T09:01:16.561 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-09T09:01:16.561 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-09T09:01:16.562 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-09T09:01:16.562 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-09T09:01:16.562 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-09T09:01:16.562 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-09T09:01:16.562 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-09T09:01:16.563 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-09T09:01:16.563 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-09T09:01:16.563 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-09T09:01:16.563 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-09T09:01:16.563 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-09T09:01:16.564 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: debug -1> 2021-03-09T09:01:16.419+0000 7fbeaaee6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fbeaaee6700 time 2021-03-09T09:01:16.419103+0000 2021-03-09T09:01:16.564 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:16.564 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:16.565 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fbeba26c1d8] 2021-03-09T09:01:16.565 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fbeba26c3f2] 2021-03-09T09:01:16.565 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x56160d37497c] 2021-03-09T09:01:16.565 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x56160d3b3ced] 2021-03-09T09:01:16.566 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x56160d3dbb5a] 2021-03-09T09:01:16.566 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x56160d3e7054] 2021-03-09T09:01:16.566 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x56160d3eb2c3] 2021-03-09T09:01:16.566 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x56160d36cc2d] 2021-03-09T09:01:16.566 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x56160d2c10b4] 2021-03-09T09:01:16.567 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 10: (C_MonOp::finish(int)+0x49) [0x56160d260069] 2021-03-09T09:01:16.567 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 11: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.567 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x56160d288da5] 2021-03-09T09:01:16.567 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 13: (Paxos::finish_round()+0x27b) [0x56160d36453b] 2021-03-09T09:01:16.568 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 14: (Paxos::commit_finish()+0x8c0) [0x56160d366f50] 2021-03-09T09:01:16.568 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 15: (C_Committed::finish(int)+0x45) [0x56160d36ab85] 2021-03-09T09:01:16.568 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-09T09:01:16.569 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-09T09:01:16.569 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-09T09:01:16.569 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-09T09:01:16.569 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-09T09:01:16.569 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-09T09:01:16.570 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.379+0000 7fe468786700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-09T09:01:16.570 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-09T09:01:16.570 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-09T09:01:16.570 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-09T09:01:16.570 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-09T09:01:16.571 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-09T09:01:16.571 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-09T09:01:16.571 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-09T09:01:16.571 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.table_properties_collectors: 2021-03-09T09:01:16.571 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.inplace_update_support: 0 2021-03-09T09:01:16.572 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-09T09:01:16.572 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-09T09:01:16.572 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-09T09:01:16.572 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-09T09:01:16.572 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.bloom_locality: 0 2021-03-09T09:01:16.573 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.max_successive_merges: 0 2021-03-09T09:01:16.573 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-09T09:01:16.573 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-09T09:01:16.573 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-09T09:01:16.574 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-09T09:01:16.574 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.ttl: 2592000 2021-03-09T09:01:16.574 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 16: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x56160d36a8a8] 2021-03-09T09:01:16.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 18: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7fbeba30baa5] 2021-03-09T09:01:16.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 20: (Thread::_entry_func(void*)+0xd) [0x7fbeba35341d] 2021-03-09T09:01:16.576 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 21: /lib64/libpthread.so.0(+0x814a) [0x7fbeb7d5214a] 2021-03-09T09:01:16.576 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 22: clone() 2021-03-09T09:01:16.576 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: debug 0> 2021-03-09T09:01:16.419+0000 7fbeaaee6700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:16.576 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: in thread 7fbeaaee6700 thread_name:fn_monstore 2021-03-09T09:01:16.576 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:16.577 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fbeb7d5cb20] 2021-03-09T09:01:16.577 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 2: gsignal() 2021-03-09T09:01:16.577 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 3: abort() 2021-03-09T09:01:16.577 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fbeba26c229] 2021-03-09T09:01:16.577 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fbeba26c3f2] 2021-03-09T09:01:16.578 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x56160d37497c] 2021-03-09T09:01:16.578 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x56160d3b3ced] 2021-03-09T09:01:16.578 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x56160d3dbb5a] 2021-03-09T09:01:16.578 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x56160d3e7054] 2021-03-09T09:01:16.579 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x56160d3eb2c3] 2021-03-09T09:01:16.579 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x56160d36cc2d] 2021-03-09T09:01:16.579 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x56160d2c10b4] 2021-03-09T09:01:16.579 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 13: (C_MonOp::finish(int)+0x49) [0x56160d260069] 2021-03-09T09:01:16.579 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 14: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.580 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-09T09:01:16.580 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: [version_set.cc:3291] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2021-03-09T09:01:16.580 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000014 succeeded,manifest_file_number is 14, next_file_number is 20, last_sequence is 2972, log_number is 13,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-09T09:01:16.581 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 13 2021-03-09T09:01:16.581 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280476384962, "job": 1, "event": "recovery_started", "log_files": [15]} 2021-03-09T09:01:16.581 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #15 mode 2 2021-03-09T09:01:16.581 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-09T09:01:16.581 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280476387069, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 20, "file_size": 15142, "table_properties": {"data_size": 14177, "index_size": 90, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 447, "raw_average_key_size": 23, "raw_value_size": 13733, "raw_average_value_size": 722, "num_data_blocks": 3, "num_entries": 19, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1615280476, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-09T09:01:16.582 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: [version_set.cc:3826] Creating manifest 21 2021-03-09T09:01:16.582 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: [version_set.cc:3291] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2021-03-09T09:01:16.582 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.383+0000 7fe468786700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280476387823, "job": 1, "event": "recovery_finished"} 2021-03-09T09:01:16.582 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.387+0000 7fe468786700 4 rocksdb: DB pointer 0x55f366705800 2021-03-09T09:01:16.583 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.387+0000 7fe457796700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-09T09:01:16.583 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.387+0000 7fe457796700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-09T09:01:16.583 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: ** DB Stats ** 2021-03-09T09:01:16.583 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:01:16.583 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-09T09:01:16.584 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-09T09:01:16.584 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x56160d288da5] 2021-03-09T09:01:16.584 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 16: (Paxos::finish_round()+0x27b) [0x56160d36453b] 2021-03-09T09:01:16.585 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 17: (Paxos::commit_finish()+0x8c0) [0x56160d366f50] 2021-03-09T09:01:16.585 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 18: (C_Committed::finish(int)+0x45) [0x56160d36ab85] 2021-03-09T09:01:16.585 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 19: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.585 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x56160d36a8a8] 2021-03-09T09:01:16.585 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 21: (Context::complete(int)+0xd) [0x56160d25d49d] 2021-03-09T09:01:16.586 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7fbeba30baa5] 2021-03-09T09:01:16.586 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 23: (Thread::_entry_func(void*)+0xd) [0x7fbeba35341d] 2021-03-09T09:01:16.586 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 24: /lib64/libpthread.so.0(+0x814a) [0x7fbeb7d5214a] 2021-03-09T09:01:16.586 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: 25: clone() 2021-03-09T09:01:16.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[25980]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:16.587 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:01:16.588 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-09T09:01:16.588 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-09T09:01:16.588 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:01:16.588 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: ** Compaction Stats [default] ** 2021-03-09T09:01:16.589 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:16.589 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:16.589 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: L0 1/0 14.79 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 8.0 0.00 0.00 1 0.002 0 0 2021-03-09T09:01:16.589 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: L6 1/0 7.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:01:16.589 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Sum 2/0 7.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 8.0 0.00 0.00 1 0.002 0 0 2021-03-09T09:01:16.590 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 8.0 0.00 0.00 1 0.002 0 0 2021-03-09T09:01:16.590 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: ** Compaction Stats [default] ** 2021-03-09T09:01:16.590 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:16.590 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:16.591 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 8.0 0.00 0.00 1 0.002 0 0 2021-03-09T09:01:16.591 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:01:16.591 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:16.591 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:16.592 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:01:16.592 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:01:16.592 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:01:16.592 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Cumulative compaction: 0.00 GB write, 2.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:16.592 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Interval compaction: 0.00 GB write, 2.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:16.593 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:01:16.593 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:01:16.593 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: ** Compaction Stats [default] ** 2021-03-09T09:01:16.593 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:16.593 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:16.594 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: L0 1/0 14.79 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 8.0 0.00 0.00 1 0.002 0 0 2021-03-09T09:01:16.594 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: L6 1/0 7.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:01:16.594 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Sum 2/0 7.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 8.0 0.00 0.00 1 0.002 0 0 2021-03-09T09:01:16.594 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:01:16.595 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: ** Compaction Stats [default] ** 2021-03-09T09:01:16.595 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:16.595 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:16.595 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 8.0 0.00 0.00 1 0.002 0 0 2021-03-09T09:01:16.595 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:01:16.596 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:16.596 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:16.596 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:01:16.596 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:01:16.596 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:01:16.597 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Cumulative compaction: 0.00 GB write, 2.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:16.597 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:16.597 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:01:16.597 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:01:16.597 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.387+0000 7fe468786700 0 starting mon.c rank 1 at public addrs [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] at bind addrs [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:01:16.598 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.387+0000 7fe468786700 1 mon.c@-1(???) e3 preinit fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:01:16.598 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.387+0000 7fe468786700 0 mon.c@-1(???).mds e1 new map 2021-03-09T09:01:16.598 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.391+0000 7fe468786700 0 mon.c@-1(???).mds e1 print_map 2021-03-09T09:01:16.598 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: e1 2021-03-09T09:01:16.599 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: enable_multiple, ever_enabled_multiple: 1,1 2021-03-09T09:01:16.599 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-03-09T09:01:16.599 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: legacy client fscid: -1 2021-03-09T09:01:16.599 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: 2021-03-09T09:01:16.599 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: No filesystems configured 2021-03-09T09:01:16.600 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.391+0000 7fe468786700 0 mon.c@-1(???).osd e18 crush map has features 3314932999778484224, adjusting msgr requires 2021-03-09T09:01:16.600 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.391+0000 7fe468786700 0 mon.c@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:01:16.600 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.391+0000 7fe468786700 0 mon.c@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:01:16.600 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.391+0000 7fe468786700 0 mon.c@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:01:16.601 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: audit 2021-03-09T08:59:48.920210+0000 mon.a (mon.0) 269 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T09:01:16.601 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: cluster 2021-03-09T08:59:48.920524+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-09T09:01:16.601 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: audit 2021-03-09T08:59:48.921424+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:01:16.601 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: audit 2021-03-09T08:59:48.930506+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:01:16.601 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.391+0000 7fe468786700 1 mon.c@-1(???).paxosservice(auth 1..12) refresh upgraded, format 0 -> 3 2021-03-09T09:01:16.602 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.395+0000 7fe468786700 20 expand_channel_meta expand map: {default=false} 2021-03-09T09:01:16.602 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.395+0000 7fe468786700 20 expand_channel_meta from 'false' to 'false' 2021-03-09T09:01:16.602 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.395+0000 7fe468786700 20 expand_channel_meta expanded map: {default=false} 2021-03-09T09:01:16.602 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.395+0000 7fe468786700 20 expand_channel_meta expand map: {default=info} 2021-03-09T09:01:16.602 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.395+0000 7fe468786700 20 expand_channel_meta from 'info' to 'info' 2021-03-09T09:01:16.603 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.395+0000 7fe468786700 20 expand_channel_meta expanded map: {default=info} 2021-03-09T09:01:16.603 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.395+0000 7fe468786700 20 expand_channel_meta expand map: {default=daemon} 2021-03-09T09:01:16.603 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.395+0000 7fe468786700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-03-09T09:01:16.603 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.395+0000 7fe468786700 20 expand_channel_meta expanded map: {default=daemon} 2021-03-09T09:01:16.604 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.395+0000 7fe468786700 20 expand_channel_meta expand map: {default=debug} 2021-03-09T09:01:16.604 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.395+0000 7fe468786700 20 expand_channel_meta from 'debug' to 'debug' 2021-03-09T09:01:16.604 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:16 smithi005 bash[26090]: debug 2021-03-09T09:01:16.395+0000 7fe468786700 20 expand_channel_meta expanded map: {default=debug} 2021-03-09T09:01:17.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:17 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Main process exited, code=exited, status=134/n/a 2021-03-09T09:01:17.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:17 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Failed with result 'exit-code'. 2021-03-09T09:01:19.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:01:19 smithi146 bash[13136]: debug 2021-03-09T09:01:19.362+0000 7fa560337700 -1 mon.b@2(electing) e3 get_health_metrics reporting 104 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:01:22.364 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fe458798700 time 2021-03-09T09:01:22.116492+0000 2021-03-09T09:01:22.364 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:22.365 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:22.365 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fe467b1e1d8] 2021-03-09T09:01:22.365 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fe467b1e3f2] 2021-03-09T09:01:22.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55f35d03797c] 2021-03-09T09:01:22.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55f35d076ced] 2021-03-09T09:01:22.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55f35d09eb5a] 2021-03-09T09:01:22.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55f35d0aa054] 2021-03-09T09:01:22.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55f35d0ae2c3] 2021-03-09T09:01:22.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55f35d02fc2d] 2021-03-09T09:01:22.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55f35cf840b4] 2021-03-09T09:01:22.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 10: (C_MonOp::finish(int)+0x49) [0x55f35cf23069] 2021-03-09T09:01:22.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 11: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55f35cf4bda5] 2021-03-09T09:01:22.368 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 13: (Paxos::finish_round()+0x27b) [0x55f35d02753b] 2021-03-09T09:01:22.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 14: (Paxos::commit_finish()+0x8c0) [0x55f35d029f50] 2021-03-09T09:01:22.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 15: (C_Committed::finish(int)+0x45) [0x55f35d02db85] 2021-03-09T09:01:22.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 16: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55f35d02d8a8] 2021-03-09T09:01:22.369 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 18: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7fe467bbdaa5] 2021-03-09T09:01:22.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 20: (Thread::_entry_func(void*)+0xd) [0x7fe467c0541d] 2021-03-09T09:01:22.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 21: /lib64/libpthread.so.0(+0x814a) [0x7fe46560414a] 2021-03-09T09:01:22.370 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 22: clone() 2021-03-09T09:01:22.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: *** Caught signal (Aborted) ** 2021-03-09T09:01:22.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: in thread 7fe458798700 thread_name:fn_monstore 2021-03-09T09:01:22.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: debug 2021-03-09T09:01:22.119+0000 7fe458798700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fe458798700 time 2021-03-09T09:01:22.116492+0000 2021-03-09T09:01:22.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:22.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:22.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fe467b1e1d8] 2021-03-09T09:01:22.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fe467b1e3f2] 2021-03-09T09:01:22.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55f35d03797c] 2021-03-09T09:01:22.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55f35d076ced] 2021-03-09T09:01:22.373 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55f35d09eb5a] 2021-03-09T09:01:22.374 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55f35d0aa054] 2021-03-09T09:01:22.375 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55f35d0ae2c3] 2021-03-09T09:01:22.375 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55f35d02fc2d] 2021-03-09T09:01:22.375 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55f35cf840b4] 2021-03-09T09:01:22.376 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 10: (C_MonOp::finish(int)+0x49) [0x55f35cf23069] 2021-03-09T09:01:22.376 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 11: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.376 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55f35cf4bda5] 2021-03-09T09:01:22.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 13: (Paxos::finish_round()+0x27b) [0x55f35d02753b] 2021-03-09T09:01:22.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 14: (Paxos::commit_finish()+0x8c0) [0x55f35d029f50] 2021-03-09T09:01:22.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 15: (C_Committed::finish(int)+0x45) [0x55f35d02db85] 2021-03-09T09:01:22.377 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 16: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55f35d02d8a8] 2021-03-09T09:01:22.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 18: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7fe467bbdaa5] 2021-03-09T09:01:22.378 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 20: (Thread::_entry_func(void*)+0xd) [0x7fe467c0541d] 2021-03-09T09:01:22.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 21: /lib64/libpthread.so.0(+0x814a) [0x7fe46560414a] 2021-03-09T09:01:22.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 22: clone() 2021-03-09T09:01:22.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:22.379 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fe46560eb20] 2021-03-09T09:01:22.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 2: gsignal() 2021-03-09T09:01:22.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 3: abort() 2021-03-09T09:01:22.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fe467b1e229] 2021-03-09T09:01:22.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fe467b1e3f2] 2021-03-09T09:01:22.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55f35d03797c] 2021-03-09T09:01:22.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55f35d076ced] 2021-03-09T09:01:22.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55f35d09eb5a] 2021-03-09T09:01:22.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55f35d0aa054] 2021-03-09T09:01:22.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55f35d0ae2c3] 2021-03-09T09:01:22.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55f35d02fc2d] 2021-03-09T09:01:22.382 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55f35cf840b4] 2021-03-09T09:01:22.382 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 13: (C_MonOp::finish(int)+0x49) [0x55f35cf23069] 2021-03-09T09:01:22.382 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 14: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.382 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55f35cf4bda5] 2021-03-09T09:01:22.383 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 16: (Paxos::finish_round()+0x27b) [0x55f35d02753b] 2021-03-09T09:01:22.383 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 17: (Paxos::commit_finish()+0x8c0) [0x55f35d029f50] 2021-03-09T09:01:22.383 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 18: (C_Committed::finish(int)+0x45) [0x55f35d02db85] 2021-03-09T09:01:22.383 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 19: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.383 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55f35d02d8a8] 2021-03-09T09:01:22.384 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 21: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.384 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7fe467bbdaa5] 2021-03-09T09:01:22.384 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 23: (Thread::_entry_func(void*)+0xd) [0x7fe467c0541d] 2021-03-09T09:01:22.384 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 24: /lib64/libpthread.so.0(+0x814a) [0x7fe46560414a] 2021-03-09T09:01:22.384 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 25: clone() 2021-03-09T09:01:22.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: debug 2021-03-09T09:01:22.127+0000 7fe458798700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:22.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: in thread 7fe458798700 thread_name:fn_monstore 2021-03-09T09:01:22.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:22.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fe46560eb20] 2021-03-09T09:01:22.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 2: gsignal() 2021-03-09T09:01:22.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 3: abort() 2021-03-09T09:01:22.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fe467b1e229] 2021-03-09T09:01:22.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fe467b1e3f2] 2021-03-09T09:01:22.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55f35d03797c] 2021-03-09T09:01:22.387 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55f35d076ced] 2021-03-09T09:01:22.387 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55f35d09eb5a] 2021-03-09T09:01:22.387 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55f35d0aa054] 2021-03-09T09:01:22.387 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55f35d0ae2c3] 2021-03-09T09:01:22.387 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55f35d02fc2d] 2021-03-09T09:01:22.388 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55f35cf840b4] 2021-03-09T09:01:22.388 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 13: (C_MonOp::finish(int)+0x49) [0x55f35cf23069] 2021-03-09T09:01:22.388 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 14: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.388 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55f35cf4bda5] 2021-03-09T09:01:22.389 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 16: (Paxos::finish_round()+0x27b) [0x55f35d02753b] 2021-03-09T09:01:22.389 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 17: (Paxos::commit_finish()+0x8c0) [0x55f35d029f50] 2021-03-09T09:01:22.389 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 18: (C_Committed::finish(int)+0x45) [0x55f35d02db85] 2021-03-09T09:01:22.389 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 19: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.389 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55f35d02d8a8] 2021-03-09T09:01:22.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 21: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7fe467bbdaa5] 2021-03-09T09:01:22.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 23: (Thread::_entry_func(void*)+0xd) [0x7fe467c0541d] 2021-03-09T09:01:22.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 24: /lib64/libpthread.so.0(+0x814a) [0x7fe46560414a] 2021-03-09T09:01:22.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 25: clone() 2021-03-09T09:01:22.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:22.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: debug -1> 2021-03-09T09:01:22.119+0000 7fe458798700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fe458798700 time 2021-03-09T09:01:22.116492+0000 2021-03-09T09:01:22.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:22.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:22.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fe467b1e1d8] 2021-03-09T09:01:22.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fe467b1e3f2] 2021-03-09T09:01:22.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55f35d03797c] 2021-03-09T09:01:22.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55f35d076ced] 2021-03-09T09:01:22.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55f35d09eb5a] 2021-03-09T09:01:22.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55f35d0aa054] 2021-03-09T09:01:22.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55f35d0ae2c3] 2021-03-09T09:01:22.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55f35d02fc2d] 2021-03-09T09:01:22.394 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55f35cf840b4] 2021-03-09T09:01:22.394 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 10: (C_MonOp::finish(int)+0x49) [0x55f35cf23069] 2021-03-09T09:01:22.394 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 11: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.395 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55f35cf4bda5] 2021-03-09T09:01:22.395 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 13: (Paxos::finish_round()+0x27b) [0x55f35d02753b] 2021-03-09T09:01:22.395 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 14: (Paxos::commit_finish()+0x8c0) [0x55f35d029f50] 2021-03-09T09:01:22.395 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 15: (C_Committed::finish(int)+0x45) [0x55f35d02db85] 2021-03-09T09:01:22.395 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 16: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.396 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55f35d02d8a8] 2021-03-09T09:01:22.396 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 18: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.396 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7fe467bbdaa5] 2021-03-09T09:01:22.396 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 20: (Thread::_entry_func(void*)+0xd) [0x7fe467c0541d] 2021-03-09T09:01:22.396 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 21: /lib64/libpthread.so.0(+0x814a) [0x7fe46560414a] 2021-03-09T09:01:22.397 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 22: clone() 2021-03-09T09:01:22.397 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: debug 0> 2021-03-09T09:01:22.127+0000 7fe458798700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:22.397 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: in thread 7fe458798700 thread_name:fn_monstore 2021-03-09T09:01:22.397 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:22.398 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fe46560eb20] 2021-03-09T09:01:22.398 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 2: gsignal() 2021-03-09T09:01:22.398 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 3: abort() 2021-03-09T09:01:22.398 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fe467b1e229] 2021-03-09T09:01:22.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fe467b1e3f2] 2021-03-09T09:01:22.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55f35d03797c] 2021-03-09T09:01:22.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55f35d076ced] 2021-03-09T09:01:22.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55f35d09eb5a] 2021-03-09T09:01:22.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55f35d0aa054] 2021-03-09T09:01:22.400 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55f35d0ae2c3] 2021-03-09T09:01:22.400 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55f35d02fc2d] 2021-03-09T09:01:22.400 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55f35cf840b4] 2021-03-09T09:01:22.400 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 13: (C_MonOp::finish(int)+0x49) [0x55f35cf23069] 2021-03-09T09:01:22.401 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 14: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.401 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55f35cf4bda5] 2021-03-09T09:01:22.401 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 16: (Paxos::finish_round()+0x27b) [0x55f35d02753b] 2021-03-09T09:01:22.401 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 17: (Paxos::commit_finish()+0x8c0) [0x55f35d029f50] 2021-03-09T09:01:22.401 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 18: (C_Committed::finish(int)+0x45) [0x55f35d02db85] 2021-03-09T09:01:22.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 19: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55f35d02d8a8] 2021-03-09T09:01:22.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 21: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7fe467bbdaa5] 2021-03-09T09:01:22.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 23: (Thread::_entry_func(void*)+0xd) [0x7fe467c0541d] 2021-03-09T09:01:22.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 24: /lib64/libpthread.so.0(+0x814a) [0x7fe46560414a] 2021-03-09T09:01:22.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 25: clone() 2021-03-09T09:01:22.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:22.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: debug -1> 2021-03-09T09:01:22.119+0000 7fe458798700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fe458798700 time 2021-03-09T09:01:22.116492+0000 2021-03-09T09:01:22.404 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:22.404 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:22.404 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fe467b1e1d8] 2021-03-09T09:01:22.404 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fe467b1e3f2] 2021-03-09T09:01:22.405 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55f35d03797c] 2021-03-09T09:01:22.405 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55f35d076ced] 2021-03-09T09:01:22.405 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55f35d09eb5a] 2021-03-09T09:01:22.405 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55f35d0aa054] 2021-03-09T09:01:22.405 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55f35d0ae2c3] 2021-03-09T09:01:22.406 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55f35d02fc2d] 2021-03-09T09:01:22.406 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55f35cf840b4] 2021-03-09T09:01:22.406 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 10: (C_MonOp::finish(int)+0x49) [0x55f35cf23069] 2021-03-09T09:01:22.407 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 11: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.407 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55f35cf4bda5] 2021-03-09T09:01:22.407 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 13: (Paxos::finish_round()+0x27b) [0x55f35d02753b] 2021-03-09T09:01:22.407 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 14: (Paxos::commit_finish()+0x8c0) [0x55f35d029f50] 2021-03-09T09:01:22.408 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 15: (C_Committed::finish(int)+0x45) [0x55f35d02db85] 2021-03-09T09:01:22.408 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 16: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.408 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55f35d02d8a8] 2021-03-09T09:01:22.408 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 18: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.408 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7fe467bbdaa5] 2021-03-09T09:01:22.409 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 20: (Thread::_entry_func(void*)+0xd) [0x7fe467c0541d] 2021-03-09T09:01:22.409 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 21: /lib64/libpthread.so.0(+0x814a) [0x7fe46560414a] 2021-03-09T09:01:22.409 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 22: clone() 2021-03-09T09:01:22.409 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: debug 0> 2021-03-09T09:01:22.127+0000 7fe458798700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:22.410 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: in thread 7fe458798700 thread_name:fn_monstore 2021-03-09T09:01:22.410 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:22.410 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fe46560eb20] 2021-03-09T09:01:22.410 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 2: gsignal() 2021-03-09T09:01:22.410 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 3: abort() 2021-03-09T09:01:22.411 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fe467b1e229] 2021-03-09T09:01:22.411 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fe467b1e3f2] 2021-03-09T09:01:22.411 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x55f35d03797c] 2021-03-09T09:01:22.411 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x55f35d076ced] 2021-03-09T09:01:22.411 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x55f35d09eb5a] 2021-03-09T09:01:22.412 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x55f35d0aa054] 2021-03-09T09:01:22.412 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x55f35d0ae2c3] 2021-03-09T09:01:22.412 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x55f35d02fc2d] 2021-03-09T09:01:22.413 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55f35cf840b4] 2021-03-09T09:01:22.413 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 13: (C_MonOp::finish(int)+0x49) [0x55f35cf23069] 2021-03-09T09:01:22.413 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 14: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.413 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x55f35cf4bda5] 2021-03-09T09:01:22.413 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 16: (Paxos::finish_round()+0x27b) [0x55f35d02753b] 2021-03-09T09:01:22.414 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 17: (Paxos::commit_finish()+0x8c0) [0x55f35d029f50] 2021-03-09T09:01:22.414 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 18: (C_Committed::finish(int)+0x45) [0x55f35d02db85] 2021-03-09T09:01:22.414 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 19: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.414 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x55f35d02d8a8] 2021-03-09T09:01:22.414 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 21: (Context::complete(int)+0xd) [0x55f35cf2049d] 2021-03-09T09:01:22.415 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7fe467bbdaa5] 2021-03-09T09:01:22.415 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 23: (Thread::_entry_func(void*)+0xd) [0x7fe467c0541d] 2021-03-09T09:01:22.415 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 24: /lib64/libpthread.so.0(+0x814a) [0x7fe46560414a] 2021-03-09T09:01:22.415 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: 25: clone() 2021-03-09T09:01:22.415 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 bash[26090]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:23.363 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Main process exited, code=exited, status=134/n/a 2021-03-09T09:01:23.364 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:22 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Failed with result 'exit-code'. 2021-03-09T09:01:24.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:01:24 smithi146 bash[13136]: debug 2021-03-09T09:01:24.362+0000 7fa560337700 -1 mon.b@2(peon) e3 get_health_metrics reporting 114 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:01:27.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:27 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Scheduled restart job, restart counter is at 4. 2021-03-09T09:01:27.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:27 smithi005 systemd[1]: Stopped Ceph mon.a for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:01:27.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:27 smithi005 systemd[1]: Started Ceph mon.a for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:01:28.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.475+0000 7f43e56a0700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-09T09:01:28.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.475+0000 7f43e56a0700 0 ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev), process ceph-mon, pid 6 2021-03-09T09:01:28.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.475+0000 7f43e56a0700 0 pidfile_write: ignore empty --pid-file 2021-03-09T09:01:28.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 0 load: jerasure load: lrc load: isa 2021-03-09T09:01:28.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: RocksDB version: 6.8.1 2021-03-09T09:01:28.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-09T09:01:28.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Compile date Mar 9 2021 2021-03-09T09:01:28.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: DB SUMMARY 2021-03-09T09:01:28.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: CURRENT file: CURRENT 2021-03-09T09:01:28.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-09T09:01:28.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: MANIFEST file: MANIFEST-000021 size: 211 Bytes 2021-03-09T09:01:28.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 2, files: 000018.sst 000020.sst 2021-03-09T09:01:28.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000022.log size: 18634 ; 2021-03-09T09:01:28.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.error_if_exists: 0 2021-03-09T09:01:28.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.create_if_missing: 0 2021-03-09T09:01:28.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.paranoid_checks: 1 2021-03-09T09:01:28.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.env: 0x5567e3e771c0 2021-03-09T09:01:28.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.fs: Posix File System 2021-03-09T09:01:28.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.info_log: 0x5567ed60b120 2021-03-09T09:01:28.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-09T09:01:28.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.statistics: (nil) 2021-03-09T09:01:28.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.use_fsync: 0 2021-03-09T09:01:28.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_log_file_size: 0 2021-03-09T09:01:28.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-09T09:01:28.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-09T09:01:28.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-09T09:01:28.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-09T09:01:28.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.allow_fallocate: 1 2021-03-09T09:01:28.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-09T09:01:28.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-09T09:01:28.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.use_direct_reads: 0 2021-03-09T09:01:28.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-09T09:01:28.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-09T09:01:28.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.db_log_dir: 2021-03-09T09:01:28.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2021-03-09T09:01:28.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-09T09:01:28.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_subcompactions: 1 2021-03-09T09:01:28.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_background_flushes: -1 2021-03-09T09:01:28.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-09T09:01:28.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-09T09:01:28.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-09T09:01:28.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-09T09:01:28.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-09T09:01:28.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-09T09:01:28.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-09T09:01:28.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.write_buffer_manager: 0x5567ed65fcb0 2021-03-09T09:01:28.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-09T09:01:28.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-09T09:01:28.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-09T09:01:28.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-09T09:01:28.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-09T09:01:28.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-09T09:01:28.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-09T09:01:28.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-09T09:01:28.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-09T09:01:28.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.unordered_write: 0 2021-03-09T09:01:28.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-09T09:01:28.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-09T09:01:28.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-09T09:01:28.630 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-09T09:01:28.630 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.row_cache: None 2021-03-09T09:01:28.630 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.wal_filter: None 2021-03-09T09:01:28.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-09T09:01:28.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-09T09:01:28.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.preserve_deletes: 0 2021-03-09T09:01:28.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.two_write_queues: 0 2021-03-09T09:01:28.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-09T09:01:28.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.atomic_flush: 0 2021-03-09T09:01:28.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-09T09:01:28.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-09T09:01:28.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-09T09:01:28.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.log_readahead_size: 0 2021-03-09T09:01:28.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-09T09:01:28.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_background_jobs: 2 2021-03-09T09:01:28.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_background_compactions: -1 2021-03-09T09:01:28.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-09T09:01:28.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-09T09:01:28.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-09T09:01:28.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-09T09:01:28.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-09T09:01:28.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-09T09:01:28.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-09T09:01:28.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-09T09:01:28.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_open_files: -1 2021-03-09T09:01:28.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-09T09:01:28.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-09T09:01:28.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-09T09:01:28.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-09T09:01:28.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Compression algorithms supported: 2021-03-09T09:01:28.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-09T09:01:28.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: kZSTD supported: 0 2021-03-09T09:01:28.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: kXpressCompression supported: 0 2021-03-09T09:01:28.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-09T09:01:28.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: kLZ4Compression supported: 1 2021-03-09T09:01:28.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: kBZip2Compression supported: 0 2021-03-09T09:01:28.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: kZlibCompression supported: 1 2021-03-09T09:01:28.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: kSnappyCompression supported: 1 2021-03-09T09:01:28.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-09T09:01:28.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000021 2021-03-09T09:01:28.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-09T09:01:28.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-09T09:01:28.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.merge_operator: 2021-03-09T09:01:28.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compaction_filter: None 2021-03-09T09:01:28.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-09T09:01:28.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-09T09:01:28.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-09T09:01:28.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5567e53ae230) 2021-03-09T09:01:28.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: cache_index_and_filter_blocks: 1 2021-03-09T09:01:28.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-09T09:01:28.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-09T09:01:28.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: pin_top_level_index_and_filter: 1 2021-03-09T09:01:28.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: index_type: 0 2021-03-09T09:01:28.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: data_block_index_type: 0 2021-03-09T09:01:28.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: index_shortening: 1 2021-03-09T09:01:28.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: data_block_hash_table_util_ratio: 0.750000 2021-03-09T09:01:28.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: hash_index_allow_collision: 1 2021-03-09T09:01:28.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: checksum: 1 2021-03-09T09:01:28.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: no_block_cache: 0 2021-03-09T09:01:28.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: block_cache: 0x5567ed5e6510 2021-03-09T09:01:28.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: block_cache_name: BinnedLRUCache 2021-03-09T09:01:28.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: block_cache_options: 2021-03-09T09:01:28.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: capacity : 536870912 2021-03-09T09:01:28.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: num_shard_bits : 4 2021-03-09T09:01:28.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: strict_capacity_limit : 0 2021-03-09T09:01:28.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: high_pri_pool_ratio: 0.000 2021-03-09T09:01:28.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: block_cache_compressed: (nil) 2021-03-09T09:01:28.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: persistent_cache: (nil) 2021-03-09T09:01:28.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: block_size: 4096 2021-03-09T09:01:28.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: block_size_deviation: 10 2021-03-09T09:01:28.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: block_restart_interval: 16 2021-03-09T09:01:28.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: index_block_restart_interval: 1 2021-03-09T09:01:28.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: metadata_block_size: 4096 2021-03-09T09:01:28.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: partition_filters: 0 2021-03-09T09:01:28.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: use_delta_encoding: 1 2021-03-09T09:01:28.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-09T09:01:28.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: whole_key_filtering: 1 2021-03-09T09:01:28.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: verify_compression: 0 2021-03-09T09:01:28.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: read_amp_bytes_per_bit: 0 2021-03-09T09:01:28.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: format_version: 2 2021-03-09T09:01:28.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: enable_index_compression: 1 2021-03-09T09:01:28.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: block_align: 0 2021-03-09T09:01:28.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-09T09:01:28.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-09T09:01:28.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compression: NoCompression 2021-03-09T09:01:28.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-09T09:01:28.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-09T09:01:28.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-09T09:01:28.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.num_levels: 7 2021-03-09T09:01:28.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-09T09:01:28.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-09T09:01:28.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-09T09:01:28.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-09T09:01:28.651 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-09T09:01:28.651 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-09T09:01:28.651 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-09T09:01:28.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:01:28.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-09T09:01:28.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-09T09:01:28.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-09T09:01:28.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-09T09:01:28.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-09T09:01:28.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:01:28.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-09T09:01:28.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-09T09:01:28.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-09T09:01:28.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-09T09:01:28.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-09T09:01:28.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-09T09:01:28.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-09T09:01:28.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-09T09:01:28.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-09T09:01:28.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-09T09:01:28.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-09T09:01:28.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-09T09:01:28.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-09T09:01:28.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-09T09:01:28.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-09T09:01:28.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-09T09:01:28.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-09T09:01:28.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-09T09:01:28.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-09T09:01:28.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-09T09:01:28.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-09T09:01:28.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-09T09:01:28.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-09T09:01:28.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-09T09:01:28.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-09T09:01:28.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-09T09:01:28.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-09T09:01:28.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-09T09:01:28.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-09T09:01:28.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-09T09:01:28.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-09T09:01:28.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-09T09:01:28.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-09T09:01:28.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.table_properties_collectors: 2021-03-09T09:01:28.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.inplace_update_support: 0 2021-03-09T09:01:28.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-09T09:01:28.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-09T09:01:28.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-09T09:01:28.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-09T09:01:28.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.bloom_locality: 0 2021-03-09T09:01:28.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.max_successive_merges: 0 2021-03-09T09:01:28.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-09T09:01:28.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-09T09:01:28.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-09T09:01:28.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-09T09:01:28.663 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.ttl: 2592000 2021-03-09T09:01:28.663 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-09T09:01:28.663 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: [version_set.cc:3291] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2021-03-09T09:01:28.663 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000021 succeeded,manifest_file_number is 21, next_file_number is 23, last_sequence is 3292, log_number is 16,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-09T09:01:28.663 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 16 2021-03-09T09:01:28.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280488487852, "job": 1, "event": "recovery_started", "log_files": [22]} 2021-03-09T09:01:28.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.483+0000 7f43e56a0700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #22 mode 2 2021-03-09T09:01:28.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.487+0000 7f43e56a0700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-09T09:01:28.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.487+0000 7f43e56a0700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280488489597, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 23, "file_size": 15141, "table_properties": {"data_size": 14177, "index_size": 89, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 447, "raw_average_key_size": 23, "raw_value_size": 13733, "raw_average_value_size": 722, "num_data_blocks": 3, "num_entries": 19, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1615280488, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-09T09:01:28.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.487+0000 7f43e56a0700 4 rocksdb: [version_set.cc:3826] Creating manifest 24 2021-03-09T09:01:28.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.487+0000 7f43e56a0700 4 rocksdb: [version_set.cc:3291] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2021-03-09T09:01:28.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.487+0000 7f43e56a0700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280488490615, "job": 1, "event": "recovery_finished"} 2021-03-09T09:01:28.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.491+0000 7f43e56a0700 4 rocksdb: DB pointer 0x5567ed6ab800 2021-03-09T09:01:28.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.491+0000 7f43d46b0700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-09T09:01:28.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.491+0000 7f43d46b0700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-09T09:01:28.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: ** DB Stats ** 2021-03-09T09:01:28.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:01:28.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-09T09:01:28.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-09T09:01:28.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:01:28.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-09T09:01:28.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-09T09:01:28.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:01:28.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: ** Compaction Stats [default] ** 2021-03-09T09:01:28.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:28.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:28.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: L0 2/0 29.45 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 12.4 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:28.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: L6 1/0 7.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:01:28.669 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Sum 3/0 7.69 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 12.4 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:28.669 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 12.4 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:28.669 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: ** Compaction Stats [default] ** 2021-03-09T09:01:28.669 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:28.670 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:28.670 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 12.4 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:28.670 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:01:28.670 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:28.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:28.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:01:28.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:01:28.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:01:28.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Cumulative compaction: 0.00 GB write, 1.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:28.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Interval compaction: 0.00 GB write, 1.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:28.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:01:28.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:01:28.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: ** Compaction Stats [default] ** 2021-03-09T09:01:28.673 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:28.673 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:28.673 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: L0 2/0 29.45 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 12.4 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:28.673 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: L6 1/0 7.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:01:28.673 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Sum 3/0 7.69 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 12.4 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:28.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:01:28.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: ** Compaction Stats [default] ** 2021-03-09T09:01:28.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:28.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:28.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 12.4 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:28.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:01:28.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:28.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:28.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:01:28.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:01:28.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:01:28.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Cumulative compaction: 0.00 GB write, 1.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:28.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:28.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:01:28.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:01:28.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.491+0000 7f43e56a0700 0 starting mon.a rank 0 at public addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] at bind addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:01:28.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 1 mon.a@-1(???) e3 preinit fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:01:28.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 0 mon.a@-1(???).mds e1 new map 2021-03-09T09:01:28.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 0 mon.a@-1(???).mds e1 print_map 2021-03-09T09:01:28.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: e1 2021-03-09T09:01:28.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: enable_multiple, ever_enabled_multiple: 1,1 2021-03-09T09:01:28.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-03-09T09:01:28.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: legacy client fscid: -1 2021-03-09T09:01:28.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: 2021-03-09T09:01:28.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: No filesystems configured 2021-03-09T09:01:28.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 0 mon.a@-1(???).osd e18 crush map has features 3314932999778484224, adjusting msgr requires 2021-03-09T09:01:28.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 0 mon.a@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:01:28.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 0 mon.a@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:01:28.680 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 0 mon.a@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:01:28.680 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: audit 2021-03-09T08:59:48.920210+0000 mon.a (mon.0) 269 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T09:01:28.680 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: cluster 2021-03-09T08:59:48.920524+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-09T09:01:28.680 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: audit 2021-03-09T08:59:48.921424+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:01:28.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: audit 2021-03-09T08:59:48.930506+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:01:28.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 1 mon.a@-1(???).paxosservice(auth 1..12) refresh upgraded, format 0 -> 3 2021-03-09T09:01:28.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 20 expand_channel_meta expand map: {default=false} 2021-03-09T09:01:28.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 20 expand_channel_meta from 'false' to 'false' 2021-03-09T09:01:28.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 20 expand_channel_meta expanded map: {default=false} 2021-03-09T09:01:28.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 20 expand_channel_meta expand map: {default=info} 2021-03-09T09:01:28.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 20 expand_channel_meta from 'info' to 'info' 2021-03-09T09:01:28.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 20 expand_channel_meta expanded map: {default=info} 2021-03-09T09:01:28.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 20 expand_channel_meta expand map: {default=daemon} 2021-03-09T09:01:28.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-03-09T09:01:28.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 20 expand_channel_meta expanded map: {default=daemon} 2021-03-09T09:01:28.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 20 expand_channel_meta expand map: {default=debug} 2021-03-09T09:01:28.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 20 expand_channel_meta from 'debug' to 'debug' 2021-03-09T09:01:28.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:28 smithi005 bash[26242]: debug 2021-03-09T09:01:28.495+0000 7f43e56a0700 20 expand_channel_meta expanded map: {default=debug} 2021-03-09T09:01:29.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:01:29 smithi146 bash[13136]: debug 2021-03-09T09:01:29.362+0000 7fa560337700 -1 mon.b@2(peon) e3 get_health_metrics reporting 126 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:01:33.613 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:33 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Scheduled restart job, restart counter is at 5. 2021-03-09T09:01:33.614 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:33 smithi005 systemd[1]: Stopped Ceph mon.c for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:01:33.614 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:33 smithi005 systemd[1]: Started Ceph mon.c for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:01:34.614 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.483+0000 7f0dc82c4700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-09T09:01:34.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.483+0000 7f0dc82c4700 0 ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev), process ceph-mon, pid 6 2021-03-09T09:01:34.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.483+0000 7f0dc82c4700 0 pidfile_write: ignore empty --pid-file 2021-03-09T09:01:34.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 0 load: jerasure load: lrc load: isa 2021-03-09T09:01:34.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: RocksDB version: 6.8.1 2021-03-09T09:01:34.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-09T09:01:34.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Compile date Mar 9 2021 2021-03-09T09:01:34.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: DB SUMMARY 2021-03-09T09:01:34.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: CURRENT file: CURRENT 2021-03-09T09:01:34.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-09T09:01:34.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: MANIFEST file: MANIFEST-000021 size: 205 Bytes 2021-03-09T09:01:34.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 2, files: 000018.sst 000020.sst 2021-03-09T09:01:34.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000022.log size: 18952 ; 2021-03-09T09:01:34.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.error_if_exists: 0 2021-03-09T09:01:34.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.create_if_missing: 0 2021-03-09T09:01:34.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.paranoid_checks: 1 2021-03-09T09:01:34.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.env: 0x56078c1a01c0 2021-03-09T09:01:34.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.fs: Posix File System 2021-03-09T09:01:34.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.info_log: 0x560796b95120 2021-03-09T09:01:34.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-09T09:01:34.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.statistics: (nil) 2021-03-09T09:01:34.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.use_fsync: 0 2021-03-09T09:01:34.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.max_log_file_size: 0 2021-03-09T09:01:34.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-09T09:01:34.621 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-09T09:01:34.621 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-09T09:01:34.621 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-09T09:01:34.622 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.allow_fallocate: 1 2021-03-09T09:01:34.622 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-09T09:01:34.622 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-09T09:01:34.623 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.use_direct_reads: 0 2021-03-09T09:01:34.623 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-09T09:01:34.623 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-09T09:01:34.623 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.db_log_dir: 2021-03-09T09:01:34.624 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2021-03-09T09:01:34.624 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-09T09:01:34.624 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.max_subcompactions: 1 2021-03-09T09:01:34.625 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.max_background_flushes: -1 2021-03-09T09:01:34.625 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-09T09:01:34.625 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-09T09:01:34.625 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-09T09:01:34.626 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-09T09:01:34.626 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-09T09:01:34.626 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-09T09:01:34.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-09T09:01:34.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.write_buffer_manager: 0x560796be9ce0 2021-03-09T09:01:34.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-09T09:01:34.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-09T09:01:34.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-09T09:01:34.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-09T09:01:34.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-09T09:01:34.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-09T09:01:34.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-09T09:01:34.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-09T09:01:34.630 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-09T09:01:34.630 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.unordered_write: 0 2021-03-09T09:01:34.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-09T09:01:34.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-09T09:01:34.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-09T09:01:34.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-09T09:01:34.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.row_cache: None 2021-03-09T09:01:34.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.wal_filter: None 2021-03-09T09:01:34.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-09T09:01:34.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-09T09:01:34.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.preserve_deletes: 0 2021-03-09T09:01:34.633 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.two_write_queues: 0 2021-03-09T09:01:34.633 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-09T09:01:34.633 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.atomic_flush: 0 2021-03-09T09:01:34.633 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-09T09:01:34.634 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-09T09:01:34.634 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-09T09:01:34.634 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.log_readahead_size: 0 2021-03-09T09:01:34.634 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-09T09:01:34.634 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.max_background_jobs: 2 2021-03-09T09:01:34.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.max_background_compactions: -1 2021-03-09T09:01:34.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-09T09:01:34.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-09T09:01:34.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-09T09:01:34.636 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-09T09:01:34.636 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-09T09:01:34.636 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-09T09:01:34.637 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-09T09:01:34.637 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-09T09:01:34.637 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.max_open_files: -1 2021-03-09T09:01:34.637 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-09T09:01:34.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-09T09:01:34.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-09T09:01:34.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-09T09:01:34.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Compression algorithms supported: 2021-03-09T09:01:34.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-09T09:01:34.639 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: kZSTD supported: 0 2021-03-09T09:01:34.639 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: kXpressCompression supported: 0 2021-03-09T09:01:34.639 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-09T09:01:34.639 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: kLZ4Compression supported: 1 2021-03-09T09:01:34.639 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: kBZip2Compression supported: 0 2021-03-09T09:01:34.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: kZlibCompression supported: 1 2021-03-09T09:01:34.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: kSnappyCompression supported: 1 2021-03-09T09:01:34.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-09T09:01:34.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.487+0000 7f0dc82c4700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000021 2021-03-09T09:01:34.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-09T09:01:34.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-09T09:01:34.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.merge_operator: 2021-03-09T09:01:34.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compaction_filter: None 2021-03-09T09:01:34.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-09T09:01:34.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-09T09:01:34.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-09T09:01:34.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56078e938230) 2021-03-09T09:01:34.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: cache_index_and_filter_blocks: 1 2021-03-09T09:01:34.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-09T09:01:34.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-09T09:01:34.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: pin_top_level_index_and_filter: 1 2021-03-09T09:01:34.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: index_type: 0 2021-03-09T09:01:34.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: data_block_index_type: 0 2021-03-09T09:01:34.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: index_shortening: 1 2021-03-09T09:01:34.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: data_block_hash_table_util_ratio: 0.750000 2021-03-09T09:01:34.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: hash_index_allow_collision: 1 2021-03-09T09:01:34.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: checksum: 1 2021-03-09T09:01:34.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: no_block_cache: 0 2021-03-09T09:01:34.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: block_cache: 0x560796b70510 2021-03-09T09:01:34.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: block_cache_name: BinnedLRUCache 2021-03-09T09:01:34.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: block_cache_options: 2021-03-09T09:01:34.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: capacity : 536870912 2021-03-09T09:01:34.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: num_shard_bits : 4 2021-03-09T09:01:34.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: strict_capacity_limit : 0 2021-03-09T09:01:34.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: high_pri_pool_ratio: 0.000 2021-03-09T09:01:34.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: block_cache_compressed: (nil) 2021-03-09T09:01:34.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: persistent_cache: (nil) 2021-03-09T09:01:34.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: block_size: 4096 2021-03-09T09:01:34.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: block_size_deviation: 10 2021-03-09T09:01:34.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: block_restart_interval: 16 2021-03-09T09:01:34.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: index_block_restart_interval: 1 2021-03-09T09:01:34.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: metadata_block_size: 4096 2021-03-09T09:01:34.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: partition_filters: 0 2021-03-09T09:01:34.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: use_delta_encoding: 1 2021-03-09T09:01:34.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-09T09:01:34.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: whole_key_filtering: 1 2021-03-09T09:01:34.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: verify_compression: 0 2021-03-09T09:01:34.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: read_amp_bytes_per_bit: 0 2021-03-09T09:01:34.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: format_version: 2 2021-03-09T09:01:34.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: enable_index_compression: 1 2021-03-09T09:01:34.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: block_align: 0 2021-03-09T09:01:34.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-09T09:01:34.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-09T09:01:34.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compression: NoCompression 2021-03-09T09:01:34.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-09T09:01:34.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-09T09:01:34.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-09T09:01:34.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.num_levels: 7 2021-03-09T09:01:34.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-09T09:01:34.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-09T09:01:34.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-09T09:01:34.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-09T09:01:34.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-09T09:01:34.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-09T09:01:34.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-09T09:01:34.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:01:34.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-09T09:01:34.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-09T09:01:34.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-09T09:01:34.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-09T09:01:34.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-09T09:01:34.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:01:34.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-09T09:01:34.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-09T09:01:34.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-09T09:01:34.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-09T09:01:34.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-09T09:01:34.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-09T09:01:34.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-09T09:01:34.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-09T09:01:34.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-09T09:01:34.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-09T09:01:34.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-09T09:01:34.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-09T09:01:34.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-09T09:01:34.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-09T09:01:34.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-09T09:01:34.658 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-09T09:01:34.658 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-09T09:01:34.658 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-09T09:01:34.658 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-09T09:01:34.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-09T09:01:34.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-09T09:01:34.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-09T09:01:34.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-09T09:01:34.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-09T09:01:34.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-09T09:01:34.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-09T09:01:34.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-09T09:01:34.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-09T09:01:34.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-09T09:01:34.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-09T09:01:34.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-09T09:01:34.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-09T09:01:34.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-09T09:01:34.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.table_properties_collectors: 2021-03-09T09:01:34.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.inplace_update_support: 0 2021-03-09T09:01:34.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-09T09:01:34.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-09T09:01:34.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-09T09:01:34.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-09T09:01:34.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.bloom_locality: 0 2021-03-09T09:01:34.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.max_successive_merges: 0 2021-03-09T09:01:34.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-09T09:01:34.664 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-09T09:01:34.664 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-09T09:01:34.664 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-09T09:01:34.664 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.ttl: 2592000 2021-03-09T09:01:34.664 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-09T09:01:34.665 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: [version_set.cc:3291] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2021-03-09T09:01:34.665 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000021 succeeded,manifest_file_number is 21, next_file_number is 23, last_sequence is 3003, log_number is 16,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-09T09:01:34.665 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 16 2021-03-09T09:01:34.665 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280494493399, "job": 1, "event": "recovery_started", "log_files": [22]} 2021-03-09T09:01:34.665 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #22 mode 2 2021-03-09T09:01:34.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-09T09:01:34.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280494495537, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 23, "file_size": 15015, "table_properties": {"data_size": 14050, "index_size": 90, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 359, "raw_average_key_size": 23, "raw_value_size": 13688, "raw_average_value_size": 912, "num_data_blocks": 3, "num_entries": 15, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1615280494, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-09T09:01:34.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: [version_set.cc:3826] Creating manifest 24 2021-03-09T09:01:34.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.491+0000 7f0dc82c4700 4 rocksdb: [version_set.cc:3291] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2021-03-09T09:01:34.667 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.495+0000 7f0dc82c4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280494496396, "job": 1, "event": "recovery_finished"} 2021-03-09T09:01:34.667 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.495+0000 7f0dc82c4700 4 rocksdb: DB pointer 0x560796c35800 2021-03-09T09:01:34.667 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.495+0000 7f0db72d4700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-09T09:01:34.667 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.495+0000 7f0db72d4700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-09T09:01:34.667 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: ** DB Stats ** 2021-03-09T09:01:34.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:01:34.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-09T09:01:34.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-09T09:01:34.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:01:34.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-09T09:01:34.669 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-09T09:01:34.669 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:01:34.669 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: ** Compaction Stats [default] ** 2021-03-09T09:01:34.669 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:34.670 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:34.670 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: L0 2/0 29.45 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 9.4 0.00 0.00 1 0.002 0 0 2021-03-09T09:01:34.670 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: L6 1/0 7.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:01:34.670 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Sum 3/0 7.70 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 9.4 0.00 0.00 1 0.002 0 0 2021-03-09T09:01:34.670 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 9.4 0.00 0.00 1 0.002 0 0 2021-03-09T09:01:34.671 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: ** Compaction Stats [default] ** 2021-03-09T09:01:34.671 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:34.671 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:34.671 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 9.4 0.00 0.00 1 0.002 0 0 2021-03-09T09:01:34.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:01:34.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:34.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:34.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:01:34.673 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:01:34.673 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:01:34.673 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Cumulative compaction: 0.00 GB write, 2.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:34.680 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Interval compaction: 0.00 GB write, 2.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:34.681 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:01:34.681 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:01:34.681 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: ** Compaction Stats [default] ** 2021-03-09T09:01:34.681 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:34.682 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:34.682 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: L0 2/0 29.45 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 9.4 0.00 0.00 1 0.002 0 0 2021-03-09T09:01:34.682 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: L6 1/0 7.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:01:34.682 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Sum 3/0 7.70 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 9.4 0.00 0.00 1 0.002 0 0 2021-03-09T09:01:34.683 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:01:34.683 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: ** Compaction Stats [default] ** 2021-03-09T09:01:34.683 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:34.683 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:34.683 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 9.4 0.00 0.00 1 0.002 0 0 2021-03-09T09:01:34.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:01:34.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:34.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:34.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:01:34.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:01:34.685 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:01:34.685 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Cumulative compaction: 0.00 GB write, 2.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:34.685 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:34.685 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:01:34.686 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:01:34.686 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.499+0000 7f0dc82c4700 0 starting mon.c rank 1 at public addrs [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] at bind addrs [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:01:34.686 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.499+0000 7f0dc82c4700 1 mon.c@-1(???) e3 preinit fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:01:34.686 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.499+0000 7f0dc82c4700 0 mon.c@-1(???).mds e1 new map 2021-03-09T09:01:34.686 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.499+0000 7f0dc82c4700 0 mon.c@-1(???).mds e1 print_map 2021-03-09T09:01:34.687 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: e1 2021-03-09T09:01:34.687 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: enable_multiple, ever_enabled_multiple: 1,1 2021-03-09T09:01:34.687 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-03-09T09:01:34.687 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: legacy client fscid: -1 2021-03-09T09:01:34.688 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: 2021-03-09T09:01:34.688 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: No filesystems configured 2021-03-09T09:01:34.688 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.499+0000 7f0dc82c4700 0 mon.c@-1(???).osd e18 crush map has features 3314932999778484224, adjusting msgr requires 2021-03-09T09:01:34.688 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.499+0000 7f0dc82c4700 0 mon.c@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:01:34.688 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.499+0000 7f0dc82c4700 0 mon.c@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:01:34.689 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.499+0000 7f0dc82c4700 0 mon.c@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:01:34.689 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: audit 2021-03-09T08:59:48.920210+0000 mon.a (mon.0) 269 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T09:01:34.689 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: cluster 2021-03-09T08:59:48.920524+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-09T09:01:34.689 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: audit 2021-03-09T08:59:48.921424+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:01:34.689 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: audit 2021-03-09T08:59:48.930506+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:01:34.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.503+0000 7f0dc82c4700 1 mon.c@-1(???).paxosservice(auth 1..12) refresh upgraded, format 0 -> 3 2021-03-09T09:01:34.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.503+0000 7f0dc82c4700 20 expand_channel_meta expand map: {default=false} 2021-03-09T09:01:34.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.503+0000 7f0dc82c4700 20 expand_channel_meta from 'false' to 'false' 2021-03-09T09:01:34.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.503+0000 7f0dc82c4700 20 expand_channel_meta expanded map: {default=false} 2021-03-09T09:01:34.691 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.503+0000 7f0dc82c4700 20 expand_channel_meta expand map: {default=info} 2021-03-09T09:01:34.691 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.503+0000 7f0dc82c4700 20 expand_channel_meta from 'info' to 'info' 2021-03-09T09:01:34.691 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.503+0000 7f0dc82c4700 20 expand_channel_meta expanded map: {default=info} 2021-03-09T09:01:34.691 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.503+0000 7f0dc82c4700 20 expand_channel_meta expand map: {default=daemon} 2021-03-09T09:01:34.691 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.503+0000 7f0dc82c4700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-03-09T09:01:34.692 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.503+0000 7f0dc82c4700 20 expand_channel_meta expanded map: {default=daemon} 2021-03-09T09:01:34.692 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.503+0000 7f0dc82c4700 20 expand_channel_meta expand map: {default=debug} 2021-03-09T09:01:34.692 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.503+0000 7f0dc82c4700 20 expand_channel_meta from 'debug' to 'debug' 2021-03-09T09:01:34.692 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:34 smithi005 bash[26354]: debug 2021-03-09T09:01:34.503+0000 7f0dc82c4700 20 expand_channel_meta expanded map: {default=debug} 2021-03-09T09:01:34.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:01:34 smithi146 bash[13136]: debug 2021-03-09T09:01:34.366+0000 7fa560337700 -1 mon.b@2(electing) e3 get_health_metrics reporting 135 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:01:35.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f43d56b2700 time 2021-03-09T09:01:35.239628+0000 2021-03-09T09:01:35.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:35.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:35.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f43e4a381d8] 2021-03-09T09:01:35.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f43e4a383f2] 2021-03-09T09:01:35.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x5567e36d997c] 2021-03-09T09:01:35.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x5567e3718ced] 2021-03-09T09:01:35.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x5567e3740b5a] 2021-03-09T09:01:35.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x5567e374c054] 2021-03-09T09:01:35.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x5567e37502c3] 2021-03-09T09:01:35.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x5567e36d1c2d] 2021-03-09T09:01:35.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x5567e36260b4] 2021-03-09T09:01:35.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 10: (C_MonOp::finish(int)+0x49) [0x5567e35c5069] 2021-03-09T09:01:35.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 11: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x5567e35edda5] 2021-03-09T09:01:35.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 13: (Paxos::finish_round()+0x27b) [0x5567e36c953b] 2021-03-09T09:01:35.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 14: (Paxos::commit_finish()+0x8c0) [0x5567e36cbf50] 2021-03-09T09:01:35.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 15: (C_Committed::finish(int)+0x45) [0x5567e36cfb85] 2021-03-09T09:01:35.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 16: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x5567e36cf8a8] 2021-03-09T09:01:35.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 18: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f43e4ad7aa5] 2021-03-09T09:01:35.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 20: (Thread::_entry_func(void*)+0xd) [0x7f43e4b1f41d] 2021-03-09T09:01:35.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f43e251e14a] 2021-03-09T09:01:35.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 22: clone() 2021-03-09T09:01:35.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: *** Caught signal (Aborted) ** 2021-03-09T09:01:35.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: in thread 7f43d56b2700 thread_name:fn_monstore 2021-03-09T09:01:35.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: debug 2021-03-09T09:01:35.243+0000 7f43d56b2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f43d56b2700 time 2021-03-09T09:01:35.239628+0000 2021-03-09T09:01:35.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:35.621 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:35.621 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f43e4a381d8] 2021-03-09T09:01:35.621 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f43e4a383f2] 2021-03-09T09:01:35.621 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x5567e36d997c] 2021-03-09T09:01:35.621 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x5567e3718ced] 2021-03-09T09:01:35.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x5567e3740b5a] 2021-03-09T09:01:35.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x5567e374c054] 2021-03-09T09:01:35.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x5567e37502c3] 2021-03-09T09:01:35.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x5567e36d1c2d] 2021-03-09T09:01:35.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x5567e36260b4] 2021-03-09T09:01:35.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 10: (C_MonOp::finish(int)+0x49) [0x5567e35c5069] 2021-03-09T09:01:35.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 11: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x5567e35edda5] 2021-03-09T09:01:35.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 13: (Paxos::finish_round()+0x27b) [0x5567e36c953b] 2021-03-09T09:01:35.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 14: (Paxos::commit_finish()+0x8c0) [0x5567e36cbf50] 2021-03-09T09:01:35.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 15: (C_Committed::finish(int)+0x45) [0x5567e36cfb85] 2021-03-09T09:01:35.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 16: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x5567e36cf8a8] 2021-03-09T09:01:35.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 18: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f43e4ad7aa5] 2021-03-09T09:01:35.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 20: (Thread::_entry_func(void*)+0xd) [0x7f43e4b1f41d] 2021-03-09T09:01:35.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f43e251e14a] 2021-03-09T09:01:35.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 22: clone() 2021-03-09T09:01:35.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:35.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f43e2528b20] 2021-03-09T09:01:35.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 2: gsignal() 2021-03-09T09:01:35.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 3: abort() 2021-03-09T09:01:35.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f43e4a38229] 2021-03-09T09:01:35.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f43e4a383f2] 2021-03-09T09:01:35.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x5567e36d997c] 2021-03-09T09:01:35.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x5567e3718ced] 2021-03-09T09:01:35.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x5567e3740b5a] 2021-03-09T09:01:35.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x5567e374c054] 2021-03-09T09:01:35.630 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x5567e37502c3] 2021-03-09T09:01:35.630 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x5567e36d1c2d] 2021-03-09T09:01:35.630 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x5567e36260b4] 2021-03-09T09:01:35.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 13: (C_MonOp::finish(int)+0x49) [0x5567e35c5069] 2021-03-09T09:01:35.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 14: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x5567e35edda5] 2021-03-09T09:01:35.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 16: (Paxos::finish_round()+0x27b) [0x5567e36c953b] 2021-03-09T09:01:35.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 17: (Paxos::commit_finish()+0x8c0) [0x5567e36cbf50] 2021-03-09T09:01:35.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 18: (C_Committed::finish(int)+0x45) [0x5567e36cfb85] 2021-03-09T09:01:35.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 19: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x5567e36cf8a8] 2021-03-09T09:01:35.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 21: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f43e4ad7aa5] 2021-03-09T09:01:35.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 23: (Thread::_entry_func(void*)+0xd) [0x7f43e4b1f41d] 2021-03-09T09:01:35.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f43e251e14a] 2021-03-09T09:01:35.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 25: clone() 2021-03-09T09:01:35.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: debug 2021-03-09T09:01:35.247+0000 7f43d56b2700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:35.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: in thread 7f43d56b2700 thread_name:fn_monstore 2021-03-09T09:01:35.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:35.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f43e2528b20] 2021-03-09T09:01:35.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 2: gsignal() 2021-03-09T09:01:35.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 3: abort() 2021-03-09T09:01:35.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f43e4a38229] 2021-03-09T09:01:35.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f43e4a383f2] 2021-03-09T09:01:35.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x5567e36d997c] 2021-03-09T09:01:35.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x5567e3718ced] 2021-03-09T09:01:35.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x5567e3740b5a] 2021-03-09T09:01:35.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x5567e374c054] 2021-03-09T09:01:35.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x5567e37502c3] 2021-03-09T09:01:35.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x5567e36d1c2d] 2021-03-09T09:01:35.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x5567e36260b4] 2021-03-09T09:01:35.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 13: (C_MonOp::finish(int)+0x49) [0x5567e35c5069] 2021-03-09T09:01:35.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 14: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x5567e35edda5] 2021-03-09T09:01:35.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 16: (Paxos::finish_round()+0x27b) [0x5567e36c953b] 2021-03-09T09:01:35.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 17: (Paxos::commit_finish()+0x8c0) [0x5567e36cbf50] 2021-03-09T09:01:35.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 18: (C_Committed::finish(int)+0x45) [0x5567e36cfb85] 2021-03-09T09:01:35.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 19: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x5567e36cf8a8] 2021-03-09T09:01:35.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 21: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f43e4ad7aa5] 2021-03-09T09:01:35.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 23: (Thread::_entry_func(void*)+0xd) [0x7f43e4b1f41d] 2021-03-09T09:01:35.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f43e251e14a] 2021-03-09T09:01:35.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 25: clone() 2021-03-09T09:01:35.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:35.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: debug -1> 2021-03-09T09:01:35.243+0000 7f43d56b2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f43d56b2700 time 2021-03-09T09:01:35.239628+0000 2021-03-09T09:01:35.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:35.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:35.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f43e4a381d8] 2021-03-09T09:01:35.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f43e4a383f2] 2021-03-09T09:01:35.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x5567e36d997c] 2021-03-09T09:01:35.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x5567e3718ced] 2021-03-09T09:01:35.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x5567e3740b5a] 2021-03-09T09:01:35.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x5567e374c054] 2021-03-09T09:01:35.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x5567e37502c3] 2021-03-09T09:01:35.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x5567e36d1c2d] 2021-03-09T09:01:35.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x5567e36260b4] 2021-03-09T09:01:35.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 10: (C_MonOp::finish(int)+0x49) [0x5567e35c5069] 2021-03-09T09:01:35.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 11: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x5567e35edda5] 2021-03-09T09:01:35.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 13: (Paxos::finish_round()+0x27b) [0x5567e36c953b] 2021-03-09T09:01:35.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 14: (Paxos::commit_finish()+0x8c0) [0x5567e36cbf50] 2021-03-09T09:01:35.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 15: (C_Committed::finish(int)+0x45) [0x5567e36cfb85] 2021-03-09T09:01:35.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 16: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x5567e36cf8a8] 2021-03-09T09:01:35.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 18: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f43e4ad7aa5] 2021-03-09T09:01:35.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 20: (Thread::_entry_func(void*)+0xd) [0x7f43e4b1f41d] 2021-03-09T09:01:35.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f43e251e14a] 2021-03-09T09:01:35.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 22: clone() 2021-03-09T09:01:35.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: debug 0> 2021-03-09T09:01:35.247+0000 7f43d56b2700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:35.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: in thread 7f43d56b2700 thread_name:fn_monstore 2021-03-09T09:01:35.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:35.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f43e2528b20] 2021-03-09T09:01:35.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 2: gsignal() 2021-03-09T09:01:35.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 3: abort() 2021-03-09T09:01:35.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f43e4a38229] 2021-03-09T09:01:35.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f43e4a383f2] 2021-03-09T09:01:35.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x5567e36d997c] 2021-03-09T09:01:35.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x5567e3718ced] 2021-03-09T09:01:35.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x5567e3740b5a] 2021-03-09T09:01:35.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x5567e374c054] 2021-03-09T09:01:35.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x5567e37502c3] 2021-03-09T09:01:35.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x5567e36d1c2d] 2021-03-09T09:01:35.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x5567e36260b4] 2021-03-09T09:01:35.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 13: (C_MonOp::finish(int)+0x49) [0x5567e35c5069] 2021-03-09T09:01:35.651 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 14: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.651 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x5567e35edda5] 2021-03-09T09:01:35.651 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 16: (Paxos::finish_round()+0x27b) [0x5567e36c953b] 2021-03-09T09:01:35.651 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 17: (Paxos::commit_finish()+0x8c0) [0x5567e36cbf50] 2021-03-09T09:01:35.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 18: (C_Committed::finish(int)+0x45) [0x5567e36cfb85] 2021-03-09T09:01:35.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 19: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x5567e36cf8a8] 2021-03-09T09:01:35.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 21: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f43e4ad7aa5] 2021-03-09T09:01:35.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 23: (Thread::_entry_func(void*)+0xd) [0x7f43e4b1f41d] 2021-03-09T09:01:35.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f43e251e14a] 2021-03-09T09:01:35.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 25: clone() 2021-03-09T09:01:35.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:35.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: debug -1> 2021-03-09T09:01:35.243+0000 7f43d56b2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f43d56b2700 time 2021-03-09T09:01:35.239628+0000 2021-03-09T09:01:35.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:35.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:35.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f43e4a381d8] 2021-03-09T09:01:35.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f43e4a383f2] 2021-03-09T09:01:35.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x5567e36d997c] 2021-03-09T09:01:35.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x5567e3718ced] 2021-03-09T09:01:35.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x5567e3740b5a] 2021-03-09T09:01:35.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x5567e374c054] 2021-03-09T09:01:35.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x5567e37502c3] 2021-03-09T09:01:35.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x5567e36d1c2d] 2021-03-09T09:01:35.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x5567e36260b4] 2021-03-09T09:01:35.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 10: (C_MonOp::finish(int)+0x49) [0x5567e35c5069] 2021-03-09T09:01:35.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 11: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x5567e35edda5] 2021-03-09T09:01:35.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 13: (Paxos::finish_round()+0x27b) [0x5567e36c953b] 2021-03-09T09:01:35.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 14: (Paxos::commit_finish()+0x8c0) [0x5567e36cbf50] 2021-03-09T09:01:35.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 15: (C_Committed::finish(int)+0x45) [0x5567e36cfb85] 2021-03-09T09:01:35.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 16: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x5567e36cf8a8] 2021-03-09T09:01:35.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 18: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f43e4ad7aa5] 2021-03-09T09:01:35.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 20: (Thread::_entry_func(void*)+0xd) [0x7f43e4b1f41d] 2021-03-09T09:01:35.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f43e251e14a] 2021-03-09T09:01:35.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 22: clone() 2021-03-09T09:01:35.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: debug 0> 2021-03-09T09:01:35.247+0000 7f43d56b2700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:35.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: in thread 7f43d56b2700 thread_name:fn_monstore 2021-03-09T09:01:35.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:35.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f43e2528b20] 2021-03-09T09:01:35.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 2: gsignal() 2021-03-09T09:01:35.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 3: abort() 2021-03-09T09:01:35.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f43e4a38229] 2021-03-09T09:01:35.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f43e4a383f2] 2021-03-09T09:01:35.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x5567e36d997c] 2021-03-09T09:01:35.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x5567e3718ced] 2021-03-09T09:01:35.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x5567e3740b5a] 2021-03-09T09:01:35.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x5567e374c054] 2021-03-09T09:01:35.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x5567e37502c3] 2021-03-09T09:01:35.663 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x5567e36d1c2d] 2021-03-09T09:01:35.663 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x5567e36260b4] 2021-03-09T09:01:35.663 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 13: (C_MonOp::finish(int)+0x49) [0x5567e35c5069] 2021-03-09T09:01:35.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 14: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x5567e35edda5] 2021-03-09T09:01:35.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 16: (Paxos::finish_round()+0x27b) [0x5567e36c953b] 2021-03-09T09:01:35.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 17: (Paxos::commit_finish()+0x8c0) [0x5567e36cbf50] 2021-03-09T09:01:35.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 18: (C_Committed::finish(int)+0x45) [0x5567e36cfb85] 2021-03-09T09:01:35.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 19: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x5567e36cf8a8] 2021-03-09T09:01:35.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 21: (Context::complete(int)+0xd) [0x5567e35c249d] 2021-03-09T09:01:35.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f43e4ad7aa5] 2021-03-09T09:01:35.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 23: (Thread::_entry_func(void*)+0xd) [0x7f43e4b1f41d] 2021-03-09T09:01:35.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f43e251e14a] 2021-03-09T09:01:35.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: 25: clone() 2021-03-09T09:01:35.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:35 smithi005 bash[26242]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:36.363 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:36 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Main process exited, code=exited, status=134/n/a 2021-03-09T09:01:36.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:36 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Failed with result 'exit-code'. 2021-03-09T09:01:39.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:01:39 smithi146 bash[13136]: debug 2021-03-09T09:01:39.366+0000 7fa560337700 -1 mon.b@2(peon) e3 get_health_metrics reporting 146 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:01:44.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:01:44 smithi146 bash[13136]: debug 2021-03-09T09:01:44.366+0000 7fa560337700 -1 mon.b@2(peon) e3 get_health_metrics reporting 156 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:01:46.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:46 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Scheduled restart job, restart counter is at 5. 2021-03-09T09:01:46.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:46 smithi005 systemd[1]: Stopped Ceph mon.a for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:01:46.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:46 smithi005 systemd[1]: Started Ceph mon.a for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:01:47.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.335+0000 7fb36d9ea700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-09T09:01:47.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.335+0000 7fb36d9ea700 0 ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev), process ceph-mon, pid 6 2021-03-09T09:01:47.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.335+0000 7fb36d9ea700 0 pidfile_write: ignore empty --pid-file 2021-03-09T09:01:47.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 0 load: jerasure load: lrc load: isa 2021-03-09T09:01:47.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: RocksDB version: 6.8.1 2021-03-09T09:01:47.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-09T09:01:47.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Compile date Mar 9 2021 2021-03-09T09:01:47.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: DB SUMMARY 2021-03-09T09:01:47.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: CURRENT file: CURRENT 2021-03-09T09:01:47.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-09T09:01:47.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: MANIFEST file: MANIFEST-000024 size: 282 Bytes 2021-03-09T09:01:47.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 3, files: 000018.sst 000020.sst 000023.sst 2021-03-09T09:01:47.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000025.log size: 18328 ; 2021-03-09T09:01:47.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.error_if_exists: 0 2021-03-09T09:01:47.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.create_if_missing: 0 2021-03-09T09:01:47.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.paranoid_checks: 1 2021-03-09T09:01:47.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.env: 0x563e5a1a21c0 2021-03-09T09:01:47.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.fs: Posix File System 2021-03-09T09:01:47.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.info_log: 0x563e64af1120 2021-03-09T09:01:47.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-09T09:01:47.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.statistics: (nil) 2021-03-09T09:01:47.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.use_fsync: 0 2021-03-09T09:01:47.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_log_file_size: 0 2021-03-09T09:01:47.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-09T09:01:47.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-09T09:01:47.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-09T09:01:47.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-09T09:01:47.621 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.allow_fallocate: 1 2021-03-09T09:01:47.621 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-09T09:01:47.621 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-09T09:01:47.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.use_direct_reads: 0 2021-03-09T09:01:47.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-09T09:01:47.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-09T09:01:47.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.db_log_dir: 2021-03-09T09:01:47.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2021-03-09T09:01:47.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-09T09:01:47.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_subcompactions: 1 2021-03-09T09:01:47.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_background_flushes: -1 2021-03-09T09:01:47.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-09T09:01:47.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-09T09:01:47.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-09T09:01:47.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-09T09:01:47.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-09T09:01:47.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-09T09:01:47.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-09T09:01:47.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.write_buffer_manager: 0x563e64b45cb0 2021-03-09T09:01:47.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-09T09:01:47.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-09T09:01:47.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-09T09:01:47.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-09T09:01:47.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-09T09:01:47.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-09T09:01:47.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-09T09:01:47.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-09T09:01:47.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-09T09:01:47.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.unordered_write: 0 2021-03-09T09:01:47.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-09T09:01:47.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-09T09:01:47.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-09T09:01:47.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-09T09:01:47.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.row_cache: None 2021-03-09T09:01:47.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.wal_filter: None 2021-03-09T09:01:47.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-09T09:01:47.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-09T09:01:47.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.preserve_deletes: 0 2021-03-09T09:01:47.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.two_write_queues: 0 2021-03-09T09:01:47.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-09T09:01:47.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.atomic_flush: 0 2021-03-09T09:01:47.630 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-09T09:01:47.630 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-09T09:01:47.630 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-09T09:01:47.630 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.log_readahead_size: 0 2021-03-09T09:01:47.630 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-09T09:01:47.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_background_jobs: 2 2021-03-09T09:01:47.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_background_compactions: -1 2021-03-09T09:01:47.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-09T09:01:47.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-09T09:01:47.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-09T09:01:47.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-09T09:01:47.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-09T09:01:47.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-09T09:01:47.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-09T09:01:47.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-09T09:01:47.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_open_files: -1 2021-03-09T09:01:47.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-09T09:01:47.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-09T09:01:47.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-09T09:01:47.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-09T09:01:47.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Compression algorithms supported: 2021-03-09T09:01:47.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-09T09:01:47.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: kZSTD supported: 0 2021-03-09T09:01:47.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: kXpressCompression supported: 0 2021-03-09T09:01:47.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-09T09:01:47.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: kLZ4Compression supported: 1 2021-03-09T09:01:47.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: kBZip2Compression supported: 0 2021-03-09T09:01:47.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: kZlibCompression supported: 1 2021-03-09T09:01:47.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: kSnappyCompression supported: 1 2021-03-09T09:01:47.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-09T09:01:47.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000024 2021-03-09T09:01:47.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-09T09:01:47.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-09T09:01:47.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.merge_operator: 2021-03-09T09:01:47.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compaction_filter: None 2021-03-09T09:01:47.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-09T09:01:47.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-09T09:01:47.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-09T09:01:47.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563e5c894230) 2021-03-09T09:01:47.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: cache_index_and_filter_blocks: 1 2021-03-09T09:01:47.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-09T09:01:47.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-09T09:01:47.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: pin_top_level_index_and_filter: 1 2021-03-09T09:01:47.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: index_type: 0 2021-03-09T09:01:47.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: data_block_index_type: 0 2021-03-09T09:01:47.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: index_shortening: 1 2021-03-09T09:01:47.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: data_block_hash_table_util_ratio: 0.750000 2021-03-09T09:01:47.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: hash_index_allow_collision: 1 2021-03-09T09:01:47.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: checksum: 1 2021-03-09T09:01:47.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: no_block_cache: 0 2021-03-09T09:01:47.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: block_cache: 0x563e64acc510 2021-03-09T09:01:47.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: block_cache_name: BinnedLRUCache 2021-03-09T09:01:47.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: block_cache_options: 2021-03-09T09:01:47.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: capacity : 536870912 2021-03-09T09:01:47.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: num_shard_bits : 4 2021-03-09T09:01:47.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: strict_capacity_limit : 0 2021-03-09T09:01:47.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: high_pri_pool_ratio: 0.000 2021-03-09T09:01:47.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: block_cache_compressed: (nil) 2021-03-09T09:01:47.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: persistent_cache: (nil) 2021-03-09T09:01:47.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: block_size: 4096 2021-03-09T09:01:47.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: block_size_deviation: 10 2021-03-09T09:01:47.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: block_restart_interval: 16 2021-03-09T09:01:47.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: index_block_restart_interval: 1 2021-03-09T09:01:47.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: metadata_block_size: 4096 2021-03-09T09:01:47.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: partition_filters: 0 2021-03-09T09:01:47.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: use_delta_encoding: 1 2021-03-09T09:01:47.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-09T09:01:47.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: whole_key_filtering: 1 2021-03-09T09:01:47.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: verify_compression: 0 2021-03-09T09:01:47.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: read_amp_bytes_per_bit: 0 2021-03-09T09:01:47.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: format_version: 2 2021-03-09T09:01:47.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: enable_index_compression: 1 2021-03-09T09:01:47.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: block_align: 0 2021-03-09T09:01:47.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-09T09:01:47.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-09T09:01:47.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compression: NoCompression 2021-03-09T09:01:47.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-09T09:01:47.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-09T09:01:47.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-09T09:01:47.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.num_levels: 7 2021-03-09T09:01:47.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-09T09:01:47.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-09T09:01:47.651 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-09T09:01:47.651 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-09T09:01:47.651 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-09T09:01:47.651 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-09T09:01:47.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-09T09:01:47.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:01:47.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-09T09:01:47.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-09T09:01:47.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-09T09:01:47.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-09T09:01:47.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-09T09:01:47.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-09T09:01:47.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-09T09:01:47.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-09T09:01:47.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-09T09:01:47.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-09T09:01:47.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-09T09:01:47.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-09T09:01:47.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-09T09:01:47.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-09T09:01:47.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-09T09:01:47.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-09T09:01:47.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-09T09:01:47.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-09T09:01:47.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-09T09:01:47.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-09T09:01:47.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-09T09:01:47.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-09T09:01:47.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-09T09:01:47.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-09T09:01:47.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-09T09:01:47.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-09T09:01:47.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-09T09:01:47.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-09T09:01:47.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-09T09:01:47.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-09T09:01:47.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-09T09:01:47.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-09T09:01:47.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-09T09:01:47.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-09T09:01:47.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-09T09:01:47.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-09T09:01:47.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-09T09:01:47.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-09T09:01:47.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-09T09:01:47.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.table_properties_collectors: 2021-03-09T09:01:47.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.inplace_update_support: 0 2021-03-09T09:01:47.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-09T09:01:47.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-09T09:01:47.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-09T09:01:47.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-09T09:01:47.663 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.bloom_locality: 0 2021-03-09T09:01:47.663 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.max_successive_merges: 0 2021-03-09T09:01:47.663 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-09T09:01:47.663 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-09T09:01:47.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-09T09:01:47.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-09T09:01:47.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.ttl: 2592000 2021-03-09T09:01:47.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-09T09:01:47.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: [version_set.cc:3291] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2021-03-09T09:01:47.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000024 succeeded,manifest_file_number is 24, next_file_number is 26, last_sequence is 3323, log_number is 23,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-09T09:01:47.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 23 2021-03-09T09:01:47.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280507346199, "job": 1, "event": "recovery_started", "log_files": [25]} 2021-03-09T09:01:47.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #25 mode 2 2021-03-09T09:01:47.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-09T09:01:47.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280507347026, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 26, "file_size": 14836, "table_properties": {"data_size": 13871, "index_size": 90, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 447, "raw_average_key_size": 23, "raw_value_size": 13427, "raw_average_value_size": 706, "num_data_blocks": 3, "num_entries": 19, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1615280507, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-09T09:01:47.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: [version_set.cc:3826] Creating manifest 27 2021-03-09T09:01:47.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.343+0000 7fb36d9ea700 4 rocksdb: [version_set.cc:3291] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2021-03-09T09:01:47.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.347+0000 7fb36d9ea700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615280507348178, "job": 1, "event": "recovery_finished"} 2021-03-09T09:01:47.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.347+0000 7fb36d9ea700 4 rocksdb: DB pointer 0x563e64b91800 2021-03-09T09:01:47.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.347+0000 7fb35c9fa700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-09T09:01:47.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.347+0000 7fb35c9fa700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-09T09:01:47.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: ** DB Stats ** 2021-03-09T09:01:47.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:01:47.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-09T09:01:47.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-09T09:01:47.669 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:01:47.669 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-09T09:01:47.669 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-09T09:01:47.669 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-09T09:01:47.670 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: ** Compaction Stats [default] ** 2021-03-09T09:01:47.670 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:47.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:47.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: L0 3/0 43.94 KB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 23.4 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:47.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: L6 1/0 7.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:01:47.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Sum 4/0 7.71 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 23.4 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:47.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 23.4 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:47.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: ** Compaction Stats [default] ** 2021-03-09T09:01:47.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:47.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:47.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 23.4 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:47.673 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:01:47.673 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:47.673 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:47.673 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:01:47.673 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:01:47.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:01:47.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Cumulative compaction: 0.00 GB write, 2.73 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:47.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Interval compaction: 0.00 GB write, 2.73 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:47.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:01:47.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:01:47.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: ** Compaction Stats [default] ** 2021-03-09T09:01:47.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:47.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:47.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: L0 3/0 43.94 KB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 23.4 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:47.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: L6 1/0 7.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:01:47.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Sum 4/0 7.71 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 23.4 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:47.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-09T09:01:47.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: ** Compaction Stats [default] ** 2021-03-09T09:01:47.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-09T09:01:47.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-09T09:01:47.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 23.4 0.00 0.00 1 0.001 0 0 2021-03-09T09:01:47.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-09T09:01:47.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:47.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-09T09:01:47.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: AddFile(Total Files): cumulative 0, interval 0 2021-03-09T09:01:47.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-09T09:01:47.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: AddFile(Keys): cumulative 0, interval 0 2021-03-09T09:01:47.680 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Cumulative compaction: 0.00 GB write, 2.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:47.680 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-09T09:01:47.680 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-09T09:01:47.680 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: ** File Read Latency Histogram By Level [default] ** 2021-03-09T09:01:47.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.351+0000 7fb36d9ea700 0 starting mon.a rank 0 at public addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] at bind addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:01:47.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.351+0000 7fb36d9ea700 1 mon.a@-1(???) e3 preinit fsid 252abbe0-80b5-11eb-907b-001a4aab830c 2021-03-09T09:01:47.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.351+0000 7fb36d9ea700 0 mon.a@-1(???).mds e1 new map 2021-03-09T09:01:47.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.351+0000 7fb36d9ea700 0 mon.a@-1(???).mds e1 print_map 2021-03-09T09:01:47.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: e1 2021-03-09T09:01:47.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: enable_multiple, ever_enabled_multiple: 1,1 2021-03-09T09:01:47.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-03-09T09:01:47.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: legacy client fscid: -1 2021-03-09T09:01:47.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: 2021-03-09T09:01:47.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: No filesystems configured 2021-03-09T09:01:47.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.351+0000 7fb36d9ea700 0 mon.a@-1(???).osd e18 crush map has features 3314932999778484224, adjusting msgr requires 2021-03-09T09:01:47.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.351+0000 7fb36d9ea700 0 mon.a@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:01:47.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.351+0000 7fb36d9ea700 0 mon.a@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:01:47.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.351+0000 7fb36d9ea700 0 mon.a@-1(???).osd e18 crush map has features 288514050185494528, adjusting msgr requires 2021-03-09T09:01:47.684 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: audit 2021-03-09T08:59:48.920210+0000 mon.a (mon.0) 269 : audit [INF] from='osd.2 v2:172.21.15.5:6809/735746167' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2021-03-09T09:01:47.684 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: cluster 2021-03-09T08:59:48.920524+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-09T09:01:47.684 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: audit 2021-03-09T08:59:48.921424+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:01:47.684 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: audit 2021-03-09T08:59:48.930506+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14144 172.21.15.5:0/2547144181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-09T09:01:47.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.355+0000 7fb36d9ea700 1 mon.a@-1(???).paxosservice(auth 1..13) refresh upgraded, format 0 -> 3 2021-03-09T09:01:47.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.355+0000 7fb36d9ea700 20 expand_channel_meta expand map: {default=false} 2021-03-09T09:01:47.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.355+0000 7fb36d9ea700 20 expand_channel_meta from 'false' to 'false' 2021-03-09T09:01:47.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.355+0000 7fb36d9ea700 20 expand_channel_meta expanded map: {default=false} 2021-03-09T09:01:47.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.355+0000 7fb36d9ea700 20 expand_channel_meta expand map: {default=info} 2021-03-09T09:01:47.686 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.355+0000 7fb36d9ea700 20 expand_channel_meta from 'info' to 'info' 2021-03-09T09:01:47.686 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.355+0000 7fb36d9ea700 20 expand_channel_meta expanded map: {default=info} 2021-03-09T09:01:47.686 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.355+0000 7fb36d9ea700 20 expand_channel_meta expand map: {default=daemon} 2021-03-09T09:01:47.686 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.355+0000 7fb36d9ea700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-03-09T09:01:47.687 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.355+0000 7fb36d9ea700 20 expand_channel_meta expanded map: {default=daemon} 2021-03-09T09:01:47.687 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.355+0000 7fb36d9ea700 20 expand_channel_meta expand map: {default=debug} 2021-03-09T09:01:47.687 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.355+0000 7fb36d9ea700 20 expand_channel_meta from 'debug' to 'debug' 2021-03-09T09:01:47.687 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:47 smithi005 bash[26491]: debug 2021-03-09T09:01:47.355+0000 7fb36d9ea700 20 expand_channel_meta expanded map: {default=debug} 2021-03-09T09:01:49.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:01:49 smithi146 bash[13136]: debug 2021-03-09T09:01:49.370+0000 7fa560337700 -1 mon.b@2(electing) e3 get_health_metrics reporting 168 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:01:50.613 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f0db82d6700 time 2021-03-09T09:01:50.293690+0000 2021-03-09T09:01:50.614 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:50.614 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:50.614 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f0dc765c1d8] 2021-03-09T09:01:50.614 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f0dc765c3f2] 2021-03-09T09:01:50.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x56078ba0297c] 2021-03-09T09:01:50.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x56078ba41ced] 2021-03-09T09:01:50.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x56078ba69b5a] 2021-03-09T09:01:50.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x56078ba75054] 2021-03-09T09:01:50.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x56078ba792c3] 2021-03-09T09:01:50.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x56078b9fac2d] 2021-03-09T09:01:50.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x56078b94f0b4] 2021-03-09T09:01:50.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 10: (C_MonOp::finish(int)+0x49) [0x56078b8ee069] 2021-03-09T09:01:50.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 11: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x56078b916da5] 2021-03-09T09:01:50.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 13: (Paxos::finish_round()+0x27b) [0x56078b9f253b] 2021-03-09T09:01:50.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 14: (Paxos::commit_finish()+0x8c0) [0x56078b9f4f50] 2021-03-09T09:01:50.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 15: (C_Committed::finish(int)+0x45) [0x56078b9f8b85] 2021-03-09T09:01:50.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 16: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x56078b9f88a8] 2021-03-09T09:01:50.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 18: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f0dc76fbaa5] 2021-03-09T09:01:50.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 20: (Thread::_entry_func(void*)+0xd) [0x7f0dc774341d] 2021-03-09T09:01:50.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f0dc514214a] 2021-03-09T09:01:50.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 22: clone() 2021-03-09T09:01:50.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: *** Caught signal (Aborted) ** 2021-03-09T09:01:50.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: in thread 7f0db82d6700 thread_name:fn_monstore 2021-03-09T09:01:50.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: debug 2021-03-09T09:01:50.295+0000 7f0db82d6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f0db82d6700 time 2021-03-09T09:01:50.293690+0000 2021-03-09T09:01:50.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:50.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:50.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f0dc765c1d8] 2021-03-09T09:01:50.621 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f0dc765c3f2] 2021-03-09T09:01:50.621 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x56078ba0297c] 2021-03-09T09:01:50.621 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x56078ba41ced] 2021-03-09T09:01:50.621 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x56078ba69b5a] 2021-03-09T09:01:50.623 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x56078ba75054] 2021-03-09T09:01:50.623 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x56078ba792c3] 2021-03-09T09:01:50.623 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x56078b9fac2d] 2021-03-09T09:01:50.624 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x56078b94f0b4] 2021-03-09T09:01:50.624 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 10: (C_MonOp::finish(int)+0x49) [0x56078b8ee069] 2021-03-09T09:01:50.624 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 11: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.624 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x56078b916da5] 2021-03-09T09:01:50.625 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 13: (Paxos::finish_round()+0x27b) [0x56078b9f253b] 2021-03-09T09:01:50.625 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 14: (Paxos::commit_finish()+0x8c0) [0x56078b9f4f50] 2021-03-09T09:01:50.625 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 15: (C_Committed::finish(int)+0x45) [0x56078b9f8b85] 2021-03-09T09:01:50.625 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 16: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.626 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x56078b9f88a8] 2021-03-09T09:01:50.626 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 18: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.626 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f0dc76fbaa5] 2021-03-09T09:01:50.626 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 20: (Thread::_entry_func(void*)+0xd) [0x7f0dc774341d] 2021-03-09T09:01:50.626 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f0dc514214a] 2021-03-09T09:01:50.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 22: clone() 2021-03-09T09:01:50.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:50.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f0dc514cb20] 2021-03-09T09:01:50.627 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 2: gsignal() 2021-03-09T09:01:50.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 3: abort() 2021-03-09T09:01:50.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f0dc765c229] 2021-03-09T09:01:50.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f0dc765c3f2] 2021-03-09T09:01:50.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x56078ba0297c] 2021-03-09T09:01:50.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x56078ba41ced] 2021-03-09T09:01:50.629 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x56078ba69b5a] 2021-03-09T09:01:50.629 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x56078ba75054] 2021-03-09T09:01:50.629 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x56078ba792c3] 2021-03-09T09:01:50.629 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x56078b9fac2d] 2021-03-09T09:01:50.630 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x56078b94f0b4] 2021-03-09T09:01:50.630 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 13: (C_MonOp::finish(int)+0x49) [0x56078b8ee069] 2021-03-09T09:01:50.630 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 14: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.630 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x56078b916da5] 2021-03-09T09:01:50.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 16: (Paxos::finish_round()+0x27b) [0x56078b9f253b] 2021-03-09T09:01:50.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 17: (Paxos::commit_finish()+0x8c0) [0x56078b9f4f50] 2021-03-09T09:01:50.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 18: (C_Committed::finish(int)+0x45) [0x56078b9f8b85] 2021-03-09T09:01:50.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 19: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x56078b9f88a8] 2021-03-09T09:01:50.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 21: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f0dc76fbaa5] 2021-03-09T09:01:50.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 23: (Thread::_entry_func(void*)+0xd) [0x7f0dc774341d] 2021-03-09T09:01:50.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f0dc514214a] 2021-03-09T09:01:50.632 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 25: clone() 2021-03-09T09:01:50.633 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: debug 2021-03-09T09:01:50.299+0000 7f0db82d6700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:50.633 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: in thread 7f0db82d6700 thread_name:fn_monstore 2021-03-09T09:01:50.633 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:50.633 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f0dc514cb20] 2021-03-09T09:01:50.634 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 2: gsignal() 2021-03-09T09:01:50.634 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 3: abort() 2021-03-09T09:01:50.634 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f0dc765c229] 2021-03-09T09:01:50.634 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f0dc765c3f2] 2021-03-09T09:01:50.634 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x56078ba0297c] 2021-03-09T09:01:50.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x56078ba41ced] 2021-03-09T09:01:50.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x56078ba69b5a] 2021-03-09T09:01:50.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x56078ba75054] 2021-03-09T09:01:50.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x56078ba792c3] 2021-03-09T09:01:50.636 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x56078b9fac2d] 2021-03-09T09:01:50.636 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x56078b94f0b4] 2021-03-09T09:01:50.636 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 13: (C_MonOp::finish(int)+0x49) [0x56078b8ee069] 2021-03-09T09:01:50.636 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 14: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.636 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x56078b916da5] 2021-03-09T09:01:50.637 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 16: (Paxos::finish_round()+0x27b) [0x56078b9f253b] 2021-03-09T09:01:50.637 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 17: (Paxos::commit_finish()+0x8c0) [0x56078b9f4f50] 2021-03-09T09:01:50.637 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 18: (C_Committed::finish(int)+0x45) [0x56078b9f8b85] 2021-03-09T09:01:50.637 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 19: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x56078b9f88a8] 2021-03-09T09:01:50.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 21: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f0dc76fbaa5] 2021-03-09T09:01:50.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 23: (Thread::_entry_func(void*)+0xd) [0x7f0dc774341d] 2021-03-09T09:01:50.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f0dc514214a] 2021-03-09T09:01:50.639 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 25: clone() 2021-03-09T09:01:50.639 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:50.639 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: debug -1> 2021-03-09T09:01:50.295+0000 7f0db82d6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f0db82d6700 time 2021-03-09T09:01:50.293690+0000 2021-03-09T09:01:50.639 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:50.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:50.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f0dc765c1d8] 2021-03-09T09:01:50.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f0dc765c3f2] 2021-03-09T09:01:50.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x56078ba0297c] 2021-03-09T09:01:50.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x56078ba41ced] 2021-03-09T09:01:50.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x56078ba69b5a] 2021-03-09T09:01:50.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x56078ba75054] 2021-03-09T09:01:50.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x56078ba792c3] 2021-03-09T09:01:50.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x56078b9fac2d] 2021-03-09T09:01:50.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x56078b94f0b4] 2021-03-09T09:01:50.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 10: (C_MonOp::finish(int)+0x49) [0x56078b8ee069] 2021-03-09T09:01:50.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 11: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x56078b916da5] 2021-03-09T09:01:50.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 13: (Paxos::finish_round()+0x27b) [0x56078b9f253b] 2021-03-09T09:01:50.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 14: (Paxos::commit_finish()+0x8c0) [0x56078b9f4f50] 2021-03-09T09:01:50.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 15: (C_Committed::finish(int)+0x45) [0x56078b9f8b85] 2021-03-09T09:01:50.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 16: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x56078b9f88a8] 2021-03-09T09:01:50.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 18: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f0dc76fbaa5] 2021-03-09T09:01:50.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 20: (Thread::_entry_func(void*)+0xd) [0x7f0dc774341d] 2021-03-09T09:01:50.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f0dc514214a] 2021-03-09T09:01:50.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 22: clone() 2021-03-09T09:01:50.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: debug 0> 2021-03-09T09:01:50.299+0000 7f0db82d6700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:50.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: in thread 7f0db82d6700 thread_name:fn_monstore 2021-03-09T09:01:50.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:50.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f0dc514cb20] 2021-03-09T09:01:50.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 2: gsignal() 2021-03-09T09:01:50.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 3: abort() 2021-03-09T09:01:50.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f0dc765c229] 2021-03-09T09:01:50.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f0dc765c3f2] 2021-03-09T09:01:50.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x56078ba0297c] 2021-03-09T09:01:50.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x56078ba41ced] 2021-03-09T09:01:50.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x56078ba69b5a] 2021-03-09T09:01:50.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x56078ba75054] 2021-03-09T09:01:50.647 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x56078ba792c3] 2021-03-09T09:01:50.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x56078b9fac2d] 2021-03-09T09:01:50.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x56078b94f0b4] 2021-03-09T09:01:50.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 13: (C_MonOp::finish(int)+0x49) [0x56078b8ee069] 2021-03-09T09:01:50.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 14: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x56078b916da5] 2021-03-09T09:01:50.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 16: (Paxos::finish_round()+0x27b) [0x56078b9f253b] 2021-03-09T09:01:50.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 17: (Paxos::commit_finish()+0x8c0) [0x56078b9f4f50] 2021-03-09T09:01:50.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 18: (C_Committed::finish(int)+0x45) [0x56078b9f8b85] 2021-03-09T09:01:50.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 19: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x56078b9f88a8] 2021-03-09T09:01:50.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 21: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f0dc76fbaa5] 2021-03-09T09:01:50.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 23: (Thread::_entry_func(void*)+0xd) [0x7f0dc774341d] 2021-03-09T09:01:50.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f0dc514214a] 2021-03-09T09:01:50.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 25: clone() 2021-03-09T09:01:50.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:50.651 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: debug -1> 2021-03-09T09:01:50.295+0000 7f0db82d6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f0db82d6700 time 2021-03-09T09:01:50.293690+0000 2021-03-09T09:01:50.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:50.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:50.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f0dc765c1d8] 2021-03-09T09:01:50.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f0dc765c3f2] 2021-03-09T09:01:50.652 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x56078ba0297c] 2021-03-09T09:01:50.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x56078ba41ced] 2021-03-09T09:01:50.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x56078ba69b5a] 2021-03-09T09:01:50.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x56078ba75054] 2021-03-09T09:01:50.653 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x56078ba792c3] 2021-03-09T09:01:50.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x56078b9fac2d] 2021-03-09T09:01:50.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x56078b94f0b4] 2021-03-09T09:01:50.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 10: (C_MonOp::finish(int)+0x49) [0x56078b8ee069] 2021-03-09T09:01:50.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 11: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x56078b916da5] 2021-03-09T09:01:50.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 13: (Paxos::finish_round()+0x27b) [0x56078b9f253b] 2021-03-09T09:01:50.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 14: (Paxos::commit_finish()+0x8c0) [0x56078b9f4f50] 2021-03-09T09:01:50.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 15: (C_Committed::finish(int)+0x45) [0x56078b9f8b85] 2021-03-09T09:01:50.655 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 16: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x56078b9f88a8] 2021-03-09T09:01:50.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 18: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7f0dc76fbaa5] 2021-03-09T09:01:50.656 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 20: (Thread::_entry_func(void*)+0xd) [0x7f0dc774341d] 2021-03-09T09:01:50.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 21: /lib64/libpthread.so.0(+0x814a) [0x7f0dc514214a] 2021-03-09T09:01:50.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 22: clone() 2021-03-09T09:01:50.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: debug 0> 2021-03-09T09:01:50.299+0000 7f0db82d6700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:50.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: in thread 7f0db82d6700 thread_name:fn_monstore 2021-03-09T09:01:50.657 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:50.658 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f0dc514cb20] 2021-03-09T09:01:50.658 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 2: gsignal() 2021-03-09T09:01:50.658 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 3: abort() 2021-03-09T09:01:50.658 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f0dc765c229] 2021-03-09T09:01:50.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f0dc765c3f2] 2021-03-09T09:01:50.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x56078ba0297c] 2021-03-09T09:01:50.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x56078ba41ced] 2021-03-09T09:01:50.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x56078ba69b5a] 2021-03-09T09:01:50.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x56078ba75054] 2021-03-09T09:01:50.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x56078ba792c3] 2021-03-09T09:01:50.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x56078b9fac2d] 2021-03-09T09:01:50.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x56078b94f0b4] 2021-03-09T09:01:50.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 13: (C_MonOp::finish(int)+0x49) [0x56078b8ee069] 2021-03-09T09:01:50.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 14: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x56078b916da5] 2021-03-09T09:01:50.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 16: (Paxos::finish_round()+0x27b) [0x56078b9f253b] 2021-03-09T09:01:50.661 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 17: (Paxos::commit_finish()+0x8c0) [0x56078b9f4f50] 2021-03-09T09:01:50.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 18: (C_Committed::finish(int)+0x45) [0x56078b9f8b85] 2021-03-09T09:01:50.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 19: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x56078b9f88a8] 2021-03-09T09:01:50.662 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 21: (Context::complete(int)+0xd) [0x56078b8eb49d] 2021-03-09T09:01:50.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7f0dc76fbaa5] 2021-03-09T09:01:50.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 23: (Thread::_entry_func(void*)+0xd) [0x7f0dc774341d] 2021-03-09T09:01:50.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 24: /lib64/libpthread.so.0(+0x814a) [0x7f0dc514214a] 2021-03-09T09:01:50.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: 25: clone() 2021-03-09T09:01:50.663 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:50 smithi005 bash[26354]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:51.613 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:51 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Main process exited, code=exited, status=134/n/a 2021-03-09T09:01:51.613 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:01:51 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Failed with result 'exit-code'. 2021-03-09T09:01:54.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:01:54 smithi146 bash[13136]: debug 2021-03-09T09:01:54.366+0000 7fa560337700 -1 mon.b@2(electing) e3 get_health_metrics reporting 177 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:01:59.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fb35d9fc700 time 2021-03-09T09:01:59.307455+0000 2021-03-09T09:01:59.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:59.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:59.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fb36cd821d8] 2021-03-09T09:01:59.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fb36cd823f2] 2021-03-09T09:01:59.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x563e59a0497c] 2021-03-09T09:01:59.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x563e59a43ced] 2021-03-09T09:01:59.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x563e59a6bb5a] 2021-03-09T09:01:59.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x563e59a77054] 2021-03-09T09:01:59.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x563e59a7b2c3] 2021-03-09T09:01:59.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x563e599fcc2d] 2021-03-09T09:01:59.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x563e599510b4] 2021-03-09T09:01:59.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 10: (C_MonOp::finish(int)+0x49) [0x563e598f0069] 2021-03-09T09:01:59.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 11: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x563e59918da5] 2021-03-09T09:01:59.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 13: (Paxos::finish_round()+0x27b) [0x563e599f453b] 2021-03-09T09:01:59.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 14: (Paxos::commit_finish()+0x8c0) [0x563e599f6f50] 2021-03-09T09:01:59.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 15: (C_Committed::finish(int)+0x45) [0x563e599fab85] 2021-03-09T09:01:59.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 16: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x563e599fa8a8] 2021-03-09T09:01:59.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 18: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7fb36ce21aa5] 2021-03-09T09:01:59.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 20: (Thread::_entry_func(void*)+0xd) [0x7fb36ce6941d] 2021-03-09T09:01:59.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 21: /lib64/libpthread.so.0(+0x814a) [0x7fb36a86814a] 2021-03-09T09:01:59.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 22: clone() 2021-03-09T09:01:59.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: *** Caught signal (Aborted) ** 2021-03-09T09:01:59.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: in thread 7fb35d9fc700 thread_name:fn_monstore 2021-03-09T09:01:59.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: debug 2021-03-09T09:01:59.311+0000 7fb35d9fc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fb35d9fc700 time 2021-03-09T09:01:59.307455+0000 2021-03-09T09:01:59.621 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:59.621 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:59.621 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fb36cd821d8] 2021-03-09T09:01:59.621 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fb36cd823f2] 2021-03-09T09:01:59.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x563e59a0497c] 2021-03-09T09:01:59.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x563e59a43ced] 2021-03-09T09:01:59.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x563e59a6bb5a] 2021-03-09T09:01:59.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x563e59a77054] 2021-03-09T09:01:59.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x563e59a7b2c3] 2021-03-09T09:01:59.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x563e599fcc2d] 2021-03-09T09:01:59.624 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x563e599510b4] 2021-03-09T09:01:59.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 10: (C_MonOp::finish(int)+0x49) [0x563e598f0069] 2021-03-09T09:01:59.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 11: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x563e59918da5] 2021-03-09T09:01:59.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 13: (Paxos::finish_round()+0x27b) [0x563e599f453b] 2021-03-09T09:01:59.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 14: (Paxos::commit_finish()+0x8c0) [0x563e599f6f50] 2021-03-09T09:01:59.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 15: (C_Committed::finish(int)+0x45) [0x563e599fab85] 2021-03-09T09:01:59.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 16: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x563e599fa8a8] 2021-03-09T09:01:59.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 18: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7fb36ce21aa5] 2021-03-09T09:01:59.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 20: (Thread::_entry_func(void*)+0xd) [0x7fb36ce6941d] 2021-03-09T09:01:59.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 21: /lib64/libpthread.so.0(+0x814a) [0x7fb36a86814a] 2021-03-09T09:01:59.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 22: clone() 2021-03-09T09:01:59.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:59.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fb36a872b20] 2021-03-09T09:01:59.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 2: gsignal() 2021-03-09T09:01:59.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 3: abort() 2021-03-09T09:01:59.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fb36cd82229] 2021-03-09T09:01:59.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fb36cd823f2] 2021-03-09T09:01:59.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x563e59a0497c] 2021-03-09T09:01:59.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x563e59a43ced] 2021-03-09T09:01:59.630 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x563e59a6bb5a] 2021-03-09T09:01:59.630 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x563e59a77054] 2021-03-09T09:01:59.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x563e59a7b2c3] 2021-03-09T09:01:59.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x563e599fcc2d] 2021-03-09T09:01:59.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x563e599510b4] 2021-03-09T09:01:59.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 13: (C_MonOp::finish(int)+0x49) [0x563e598f0069] 2021-03-09T09:01:59.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 14: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x563e59918da5] 2021-03-09T09:01:59.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 16: (Paxos::finish_round()+0x27b) [0x563e599f453b] 2021-03-09T09:01:59.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 17: (Paxos::commit_finish()+0x8c0) [0x563e599f6f50] 2021-03-09T09:01:59.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 18: (C_Committed::finish(int)+0x45) [0x563e599fab85] 2021-03-09T09:01:59.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 19: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x563e599fa8a8] 2021-03-09T09:01:59.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 21: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7fb36ce21aa5] 2021-03-09T09:01:59.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 23: (Thread::_entry_func(void*)+0xd) [0x7fb36ce6941d] 2021-03-09T09:01:59.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 24: /lib64/libpthread.so.0(+0x814a) [0x7fb36a86814a] 2021-03-09T09:01:59.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 25: clone() 2021-03-09T09:01:59.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: debug 2021-03-09T09:01:59.315+0000 7fb35d9fc700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:59.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: in thread 7fb35d9fc700 thread_name:fn_monstore 2021-03-09T09:01:59.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:59.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fb36a872b20] 2021-03-09T09:01:59.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 2: gsignal() 2021-03-09T09:01:59.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 3: abort() 2021-03-09T09:01:59.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fb36cd82229] 2021-03-09T09:01:59.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fb36cd823f2] 2021-03-09T09:01:59.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x563e59a0497c] 2021-03-09T09:01:59.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x563e59a43ced] 2021-03-09T09:01:59.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x563e59a6bb5a] 2021-03-09T09:01:59.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x563e59a77054] 2021-03-09T09:01:59.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x563e59a7b2c3] 2021-03-09T09:01:59.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x563e599fcc2d] 2021-03-09T09:01:59.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x563e599510b4] 2021-03-09T09:01:59.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 13: (C_MonOp::finish(int)+0x49) [0x563e598f0069] 2021-03-09T09:01:59.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 14: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x563e59918da5] 2021-03-09T09:01:59.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 16: (Paxos::finish_round()+0x27b) [0x563e599f453b] 2021-03-09T09:01:59.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 17: (Paxos::commit_finish()+0x8c0) [0x563e599f6f50] 2021-03-09T09:01:59.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 18: (C_Committed::finish(int)+0x45) [0x563e599fab85] 2021-03-09T09:01:59.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 19: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x563e599fa8a8] 2021-03-09T09:01:59.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 21: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7fb36ce21aa5] 2021-03-09T09:01:59.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 23: (Thread::_entry_func(void*)+0xd) [0x7fb36ce6941d] 2021-03-09T09:01:59.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 24: /lib64/libpthread.so.0(+0x814a) [0x7fb36a86814a] 2021-03-09T09:01:59.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 25: clone() 2021-03-09T09:01:59.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:59.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: debug -1> 2021-03-09T09:01:59.311+0000 7fb35d9fc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fb35d9fc700 time 2021-03-09T09:01:59.307455+0000 2021-03-09T09:01:59.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:59.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:59.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fb36cd821d8] 2021-03-09T09:01:59.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fb36cd823f2] 2021-03-09T09:01:59.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x563e59a0497c] 2021-03-09T09:01:59.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x563e59a43ced] 2021-03-09T09:01:59.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x563e59a6bb5a] 2021-03-09T09:01:59.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x563e59a77054] 2021-03-09T09:01:59.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x563e59a7b2c3] 2021-03-09T09:01:59.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x563e599fcc2d] 2021-03-09T09:01:59.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x563e599510b4] 2021-03-09T09:01:59.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 10: (C_MonOp::finish(int)+0x49) [0x563e598f0069] 2021-03-09T09:01:59.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 11: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x563e59918da5] 2021-03-09T09:01:59.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 13: (Paxos::finish_round()+0x27b) [0x563e599f453b] 2021-03-09T09:01:59.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 14: (Paxos::commit_finish()+0x8c0) [0x563e599f6f50] 2021-03-09T09:01:59.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 15: (C_Committed::finish(int)+0x45) [0x563e599fab85] 2021-03-09T09:01:59.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 16: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x563e599fa8a8] 2021-03-09T09:01:59.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 18: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7fb36ce21aa5] 2021-03-09T09:01:59.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 20: (Thread::_entry_func(void*)+0xd) [0x7fb36ce6941d] 2021-03-09T09:01:59.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 21: /lib64/libpthread.so.0(+0x814a) [0x7fb36a86814a] 2021-03-09T09:01:59.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 22: clone() 2021-03-09T09:01:59.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: debug 0> 2021-03-09T09:01:59.315+0000 7fb35d9fc700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:59.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: in thread 7fb35d9fc700 thread_name:fn_monstore 2021-03-09T09:01:59.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:59.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fb36a872b20] 2021-03-09T09:01:59.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 2: gsignal() 2021-03-09T09:01:59.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 3: abort() 2021-03-09T09:01:59.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fb36cd82229] 2021-03-09T09:01:59.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fb36cd823f2] 2021-03-09T09:01:59.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x563e59a0497c] 2021-03-09T09:01:59.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x563e59a43ced] 2021-03-09T09:01:59.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x563e59a6bb5a] 2021-03-09T09:01:59.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x563e59a77054] 2021-03-09T09:01:59.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x563e59a7b2c3] 2021-03-09T09:01:59.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x563e599fcc2d] 2021-03-09T09:01:59.651 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x563e599510b4] 2021-03-09T09:01:59.651 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 13: (C_MonOp::finish(int)+0x49) [0x563e598f0069] 2021-03-09T09:01:59.651 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 14: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.651 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x563e59918da5] 2021-03-09T09:01:59.651 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 16: (Paxos::finish_round()+0x27b) [0x563e599f453b] 2021-03-09T09:01:59.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 17: (Paxos::commit_finish()+0x8c0) [0x563e599f6f50] 2021-03-09T09:01:59.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 18: (C_Committed::finish(int)+0x45) [0x563e599fab85] 2021-03-09T09:01:59.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 19: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x563e599fa8a8] 2021-03-09T09:01:59.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 21: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7fb36ce21aa5] 2021-03-09T09:01:59.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 23: (Thread::_entry_func(void*)+0xd) [0x7fb36ce6941d] 2021-03-09T09:01:59.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 24: /lib64/libpthread.so.0(+0x814a) [0x7fb36a86814a] 2021-03-09T09:01:59.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 25: clone() 2021-03-09T09:01:59.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:59.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: debug -1> 2021-03-09T09:01:59.311+0000 7fb35d9fc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fb35d9fc700 time 2021-03-09T09:01:59.307455+0000 2021-03-09T09:01:59.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:01:59.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:59.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fb36cd821d8] 2021-03-09T09:01:59.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fb36cd823f2] 2021-03-09T09:01:59.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x563e59a0497c] 2021-03-09T09:01:59.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x563e59a43ced] 2021-03-09T09:01:59.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x563e59a6bb5a] 2021-03-09T09:01:59.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x563e59a77054] 2021-03-09T09:01:59.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x563e59a7b2c3] 2021-03-09T09:01:59.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x563e599fcc2d] 2021-03-09T09:01:59.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x563e599510b4] 2021-03-09T09:01:59.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 10: (C_MonOp::finish(int)+0x49) [0x563e598f0069] 2021-03-09T09:01:59.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 11: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x563e59918da5] 2021-03-09T09:01:59.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 13: (Paxos::finish_round()+0x27b) [0x563e599f453b] 2021-03-09T09:01:59.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 14: (Paxos::commit_finish()+0x8c0) [0x563e599f6f50] 2021-03-09T09:01:59.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 15: (C_Committed::finish(int)+0x45) [0x563e599fab85] 2021-03-09T09:01:59.661 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 16: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x563e599fa8a8] 2021-03-09T09:01:59.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 18: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.663 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 19: (Finisher::finisher_thread_entry()+0x1a5) [0x7fb36ce21aa5] 2021-03-09T09:01:59.663 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 20: (Thread::_entry_func(void*)+0xd) [0x7fb36ce6941d] 2021-03-09T09:01:59.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 21: /lib64/libpthread.so.0(+0x814a) [0x7fb36a86814a] 2021-03-09T09:01:59.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 22: clone() 2021-03-09T09:01:59.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: debug 0> 2021-03-09T09:01:59.315+0000 7fb35d9fc700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:01:59.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: in thread 7fb35d9fc700 thread_name:fn_monstore 2021-03-09T09:01:59.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:01:59.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fb36a872b20] 2021-03-09T09:01:59.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 2: gsignal() 2021-03-09T09:01:59.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 3: abort() 2021-03-09T09:01:59.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fb36cd82229] 2021-03-09T09:01:59.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7fb36cd823f2] 2021-03-09T09:01:59.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x563e59a0497c] 2021-03-09T09:01:59.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x563e59a43ced] 2021-03-09T09:01:59.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x563e59a6bb5a] 2021-03-09T09:01:59.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x563e59a77054] 2021-03-09T09:01:59.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x563e59a7b2c3] 2021-03-09T09:01:59.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x563e599fcc2d] 2021-03-09T09:01:59.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x563e599510b4] 2021-03-09T09:01:59.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 13: (C_MonOp::finish(int)+0x49) [0x563e598f0069] 2021-03-09T09:01:59.669 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 14: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.669 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa5) [0x563e59918da5] 2021-03-09T09:01:59.669 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 16: (Paxos::finish_round()+0x27b) [0x563e599f453b] 2021-03-09T09:01:59.669 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 17: (Paxos::commit_finish()+0x8c0) [0x563e599f6f50] 2021-03-09T09:01:59.670 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 18: (C_Committed::finish(int)+0x45) [0x563e599fab85] 2021-03-09T09:01:59.670 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 19: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.670 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x563e599fa8a8] 2021-03-09T09:01:59.670 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 21: (Context::complete(int)+0xd) [0x563e598ed49d] 2021-03-09T09:01:59.670 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 22: (Finisher::finisher_thread_entry()+0x1a5) [0x7fb36ce21aa5] 2021-03-09T09:01:59.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 23: (Thread::_entry_func(void*)+0xd) [0x7fb36ce6941d] 2021-03-09T09:01:59.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 24: /lib64/libpthread.so.0(+0x814a) [0x7fb36a86814a] 2021-03-09T09:01:59.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: 25: clone() 2021-03-09T09:01:59.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:01:59 smithi005 bash[26491]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:01:59.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:01:59 smithi146 bash[13136]: debug 2021-03-09T09:01:59.370+0000 7fa560337700 -1 mon.b@2(peon) e3 get_health_metrics reporting 188 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:02:00.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:02:00 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Main process exited, code=exited, status=134/n/a 2021-03-09T09:02:00.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:02:00 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Failed with result 'exit-code'. 2021-03-09T09:02:01.863 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:02:01 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Scheduled restart job, restart counter is at 6. 2021-03-09T09:02:01.863 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:02:01 smithi005 systemd[1]: Stopped Ceph mon.c for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:02:01.864 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:02:01 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Start request repeated too quickly. 2021-03-09T09:02:01.864 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:02:01 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.c.service: Failed with result 'exit-code'. 2021-03-09T09:02:01.864 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 09 09:02:01 smithi005 systemd[1]: Failed to start Ceph mon.c for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:02:04.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:02:04 smithi146 bash[13136]: debug 2021-03-09T09:02:04.370+0000 7fa560337700 -1 mon.b@2(peon) e3 get_health_metrics reporting 198 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:02:09.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:02:09 smithi146 bash[13136]: debug 2021-03-09T09:02:09.373+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 209 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:02:10.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:02:10 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Scheduled restart job, restart counter is at 6. 2021-03-09T09:02:10.865 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:02:10 smithi005 systemd[1]: Stopped Ceph mon.a for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:02:10.865 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:02:10 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Start request repeated too quickly. 2021-03-09T09:02:10.866 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:02:10 smithi005 systemd[1]: ceph-252abbe0-80b5-11eb-907b-001a4aab830c@mon.a.service: Failed with result 'exit-code'. 2021-03-09T09:02:10.866 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 09 09:02:10 smithi005 systemd[1]: Failed to start Ceph mon.a for 252abbe0-80b5-11eb-907b-001a4aab830c. 2021-03-09T09:02:14.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:02:14 smithi146 bash[13136]: debug 2021-03-09T09:02:14.373+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 218 slow ops, oldest is log(1 entries from seq 2 at 2021-03-09T08:59:48.196146+0000) 2021-03-09T09:02:19.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338067750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:02:19.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338067750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:19.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338067750 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f833c006eb0 tx=0x7f834c05dff0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:19.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338067750 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f833c006eb0 tx=0x7f834c05dff0).stop 2021-03-09T09:02:19.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338067750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:19.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338067750 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:19.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:19.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:02:19.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:19.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338062790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:19.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338062790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:02:19.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338062790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:19.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:19.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802f010 con 0x7f834c0f4690 2021-03-09T09:02:19.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003390 con 0x7f834c0f6280 2021-03-09T09:02:19.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c420 con 0x7f8338062ca0 2021-03-09T09:02:19.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:19.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:19.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:19.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338062790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:19.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338062790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:19.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802bf70 con 0x7f834c0f6280 2021-03-09T09:02:19.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806e790 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f833c012720 tx=0x7f833c072000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:19.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c017430 con 0x7f834c0f6280 2021-03-09T09:02:19.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.376+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04e520 con 0x7f834c0f6280 2021-03-09T09:02:19.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:19.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c061070 con 0x7f834c0f6280 2021-03-09T09:02:19.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:02:19 smithi146 bash[13136]: debug 2021-03-09T09:02:19.372+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:02:24.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.375+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:02:24.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.375+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:24.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.375+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806e790 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f833c012720 tx=0x7f833c072000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:24.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.375+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806e790 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f833c012720 tx=0x7f833c072000).stop 2021-03-09T09:02:24.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.375+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806e790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:24.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.375+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806e790 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:24.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.375+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:24.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.375+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:24.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.375+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:24.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.375+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:02:24.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.375+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008eb0 con 0x7f834c0f4690 2021-03-09T09:02:24.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.375+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:24.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.375+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:02:24.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.375+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802b840 con 0x7f834c0f5930 2021-03-09T09:02:24.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.375+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:24.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.375+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806f170 con 0x7f8338062ca0 2021-03-09T09:02:24.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.379+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:24.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.379+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:24.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.379+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:24.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.379+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:24.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.379+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:24.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.379+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806f2b0 con 0x7f834c0f5930 2021-03-09T09:02:24.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.383+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833806ec80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344004e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:24.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344072030 con 0x7f834c0f5930 2021-03-09T09:02:24.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404c020 con 0x7f834c0f5930 2021-03-09T09:02:24.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:24.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404cd70 con 0x7f834c0f5930 2021-03-09T09:02:24.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:02:24 smithi146 bash[13136]: debug 2021-03-09T09:02:24.376+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:02:29.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.375+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:02:29.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.375+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:29.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.375+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833806ec80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344004e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:29.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.375+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833806ec80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344004e60).stop 2021-03-09T09:02:29.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.375+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833806ec80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:29.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.375+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833806ec80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:29.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.375+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:29.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.375+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:02:29.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.379+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:29.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.379+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380275b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:29.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.379+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:29.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.379+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067b20 con 0x7f834c0f4690 2021-03-09T09:02:29.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.379+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380605a0 con 0x7f834c0f6280 2021-03-09T09:02:29.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.379+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008d60 con 0x7f8338062ca0 2021-03-09T09:02:29.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.379+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380275b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:29.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.379+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:02:29.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.379+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:29.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.379+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:29.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.379+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:29.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.379+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:29.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.379+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:29.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.379+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806eaf0 con 0x7f834c0f6280 2021-03-09T09:02:29.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.383+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380275b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f83440696a0 tx=0x7f8344004e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:29.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834406c570 con 0x7f834c0f6280 2021-03-09T09:02:29.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344048a00 con 0x7f834c0f6280 2021-03-09T09:02:29.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:29.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834406bac0 con 0x7f834c0f6280 2021-03-09T09:02:29.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:02:29 smithi146 bash[13136]: debug 2021-03-09T09:02:29.376+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:02:34.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.374+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380275b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:02:34.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.374+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380275b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:34.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.374+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380275b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f83440696a0 tx=0x7f8344004e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:34.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.374+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380275b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f83440696a0 tx=0x7f8344004e60).stop 2021-03-09T09:02:34.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.374+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380275b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:34.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.374+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380275b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:34.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380256c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:34.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:34.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:34.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062af0 con 0x7f834c0f4690 2021-03-09T09:02:34.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e960 con 0x7f834c0f5930 2021-03-09T09:02:34.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025bb0 con 0x7f8338062ca0 2021-03-09T09:02:34.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:02:34.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:34.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:02:34.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:34.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380256c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:34.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:34.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:34.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:34.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:34.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338025cf0 con 0x7f834c0f5930 2021-03-09T09:02:34.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380256c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f8334009b30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:34.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334018070 con 0x7f834c0f5930 2021-03-09T09:02:34.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334019040 con 0x7f834c0f5930 2021-03-09T09:02:34.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:34.382+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334013410 con 0x7f834c0f5930 2021-03-09T09:02:34.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:02:34 smithi146 bash[13136]: debug 2021-03-09T09:02:34.375+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:02:39.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f83380256c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:02:39.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f83380256c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:39.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380256c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f8334009b30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:39.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380256c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f8334009b30).stop 2021-03-09T09:02:39.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f83380256c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:39.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380256c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:39.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:39.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:02:39.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:39.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:39.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:39.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006420 con 0x7f834c0f4690 2021-03-09T09:02:39.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338059890 con 0x7f834c0f6280 2021-03-09T09:02:39.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380223b0 con 0x7f8338062ca0 2021-03-09T09:02:39.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:02:39.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:39.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:39.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:39.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:39.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:39.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:39.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.378+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380224f0 con 0x7f834c0f6280 2021-03-09T09:02:39.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.382+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021ec0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f833c05bc80 tx=0x7f833c072000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:39.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.382+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04e710 con 0x7f834c0f6280 2021-03-09T09:02:39.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.382+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0208f0 con 0x7f834c0f6280 2021-03-09T09:02:39.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:39.382+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0124c0 con 0x7f834c0f6280 2021-03-09T09:02:39.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:02:39 smithi146 bash[13136]: debug 2021-03-09T09:02:39.375+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 39 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:02:44.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338021ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:02:44.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338021ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:44.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021ec0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f833c05bc80 tx=0x7f833c072000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:44.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021ec0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f833c05bc80 tx=0x7f833c072000).stop 2021-03-09T09:02:44.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338021ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:44.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021ec0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:44.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338020a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:44.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:44.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:02:44.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:44.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:44.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380223b0 con 0x7f834c0f4690 2021-03-09T09:02:44.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802eed0 con 0x7f834c0f5930 2021-03-09T09:02:44.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067940 con 0x7f8338062ca0 2021-03-09T09:02:44.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:02:44.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:44.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.378+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338020a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:44.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.382+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:44.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.382+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:44.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.382+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:44.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.382+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:44.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.382+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020ef0 con 0x7f834c0f5930 2021-03-09T09:02:44.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.382+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338020a00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f833400bdb0 tx=0x7f8334009b30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:44.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.382+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e040 con 0x7f834c0f5930 2021-03-09T09:02:44.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.382+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334048130 con 0x7f834c0f5930 2021-03-09T09:02:44.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:44.386+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334043d50 con 0x7f834c0f5930 2021-03-09T09:02:44.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:02:44 smithi146 bash[13136]: debug 2021-03-09T09:02:44.379+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 48 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:02:49.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338020a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:02:49.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338020a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:49.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338020a00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f833400bdb0 tx=0x7f8334009b30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:49.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338020a00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f833400bdb0 tx=0x7f8334009b30).stop 2021-03-09T09:02:49.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338020a00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:49.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338020a00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:49.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:49.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:49.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:02:49.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:49.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:49.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:49.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338029180 con 0x7f834c0f4690 2021-03-09T09:02:49.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338057500 con 0x7f834c0f6280 2021-03-09T09:02:49.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008b20 con 0x7f8338062ca0 2021-03-09T09:02:49.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:02:49.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.377+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:49.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.381+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:49.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.381+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:49.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.381+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:49.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.381+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:49.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.381+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380194c0 con 0x7f834c0f6280 2021-03-09T09:02:49.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.381+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018fd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f833400ae80 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:49.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.381+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e030 con 0x7f834c0f6280 2021-03-09T09:02:49.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.381+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334048310 con 0x7f834c0f6280 2021-03-09T09:02:49.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:49.385+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833406cbf0 con 0x7f834c0f6280 2021-03-09T09:02:49.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:02:49 smithi146 bash[13136]: debug 2021-03-09T09:02:49.379+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 58 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:02:54.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338018fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:02:54.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338018fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:54.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018fd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f833400ae80 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:54.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018fd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f833400ae80 tx=0x7f83340482a0).stop 2021-03-09T09:02:54.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338018fd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:54.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018fd0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:54.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:54.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:54.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338011d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:54.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026f00 con 0x7f834c0f4690 2021-03-09T09:02:54.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020d70 con 0x7f834c0f5930 2021-03-09T09:02:54.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380540b0 con 0x7f8338062ca0 2021-03-09T09:02:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:02:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:02:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:54.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338011d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:54.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:54.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:54.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:54.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ee80 con 0x7f834c0f5930 2021-03-09T09:02:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338011d40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f833c04a4c0 tx=0x7f833c072000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f834c0f5930 2021-03-09T09:02:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.381+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05c210 con 0x7f834c0f5930 2021-03-09T09:02:54.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:54.385+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04de90 con 0x7f834c0f5930 2021-03-09T09:02:54.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:02:54 smithi146 bash[13136]: debug 2021-03-09T09:02:54.378+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 82 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:02:59.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.377+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338011d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:02:59.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.377+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338011d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:59.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.377+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338011d40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f833c04a4c0 tx=0x7f833c072000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.377+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338011d40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f833c04a4c0 tx=0x7f833c072000).stop 2021-03-09T09:02:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.377+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338011d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.377+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338011d40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.377+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:59.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.377+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:59.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.377+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:02:59.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.377+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:59.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.377+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:59.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.377+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338056eb0 con 0x7f834c0f4690 2021-03-09T09:02:59.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.377+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380217f0 con 0x7f834c0f6280 2021-03-09T09:02:59.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.377+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062970 con 0x7f8338062ca0 2021-03-09T09:02:59.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.377+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:02:59.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.381+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:59.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.381+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:59.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.381+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:59.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.381+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:59.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.381+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:59.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.381+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:59.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.381+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380540b0 con 0x7f834c0f6280 2021-03-09T09:02:59.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.381+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015550 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f8334025120 tx=0x7f833400b6c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:59.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.381+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340284d0 con 0x7f834c0f6280 2021-03-09T09:02:59.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.385+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340284d0 con 0x7f834c0f6280 2021-03-09T09:02:59.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:02:59.385+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334024d30 con 0x7f834c0f6280 2021-03-09T09:02:59.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:02:59 smithi146 bash[13136]: debug 2021-03-09T09:02:59.378+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 107 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:03:04.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338015550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:03:04.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338015550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:04.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015550 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f8334025120 tx=0x7f833400b6c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:04.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015550 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f8334025120 tx=0x7f833400b6c0).stop 2021-03-09T09:03:04.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338015550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:04.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015550 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:04.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:04.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:04.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:03:04.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:04.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:03:04.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:04.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833806ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:04.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338056eb0 con 0x7f834c0f4690 2021-03-09T09:03:04.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007bb0 con 0x7f834c0f5930 2021-03-09T09:03:04.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338029170 con 0x7f8338062ca0 2021-03-09T09:03:04.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.381+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833806ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:04.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.385+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:04.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.385+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:04.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.385+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:04.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.385+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:04.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.385+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bf10 con 0x7f834c0f5930 2021-03-09T09:03:04.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.385+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833806ba20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f833c046a60 tx=0x7f833c072000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:04.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.385+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f834c0f5930 2021-03-09T09:03:04.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.389+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c004470 con 0x7f834c0f5930 2021-03-09T09:03:04.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:04.389+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f834c0f5930 2021-03-09T09:03:04.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:03:04 smithi146 bash[13136]: debug 2021-03-09T09:03:04.382+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 131 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:03:09.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833806ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:03:09.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833806ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:09.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833806ba20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f833c046a60 tx=0x7f833c072000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:09.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833806ba20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f833c046a60 tx=0x7f833c072000).stop 2021-03-09T09:03:09.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833806ba20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:09.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833806ba20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:09.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:09.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:09.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:03:09.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:09.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:09.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338056eb0 con 0x7f834c0f4690 2021-03-09T09:03:09.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ed40 con 0x7f834c0f6280 2021-03-09T09:03:09.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007b90 con 0x7f8338062ca0 2021-03-09T09:03:09.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:03:09.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:09.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:09.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:09.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:09.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:09.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:09.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a8d0 con 0x7f834c0f6280 2021-03-09T09:03:09.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806a3e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f833c04eaa0 tx=0x7f833c072000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:09.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006c20 con 0x7f834c0f6280 2021-03-09T09:03:09.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c006c20 con 0x7f834c0f6280 2021-03-09T09:03:09.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:09.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00bd30 con 0x7f834c0f6280 2021-03-09T09:03:09.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:03:09 smithi146 bash[13136]: debug 2021-03-09T09:03:09.382+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 156 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:03:14.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:03:14.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:14.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806a3e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f833c04eaa0 tx=0x7f833c072000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:14.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806a3e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f833c04eaa0 tx=0x7f833c072000).stop 2021-03-09T09:03:14.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806a3e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:14.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806a3e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:14.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:14.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338011d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:14.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:03:14.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:14.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:14.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833804f350 con 0x7f834c0f4690 2021-03-09T09:03:14.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e340 con 0x7f834c0f5930 2021-03-09T09:03:14.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380543e0 con 0x7f8338062ca0 2021-03-09T09:03:14.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:03:14.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:14.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338011d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.384+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.384+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.384+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.384+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.384+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022fb0 con 0x7f834c0f5930 2021-03-09T09:03:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.384+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338011d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f8344020d00 tx=0x7f83440502f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344048830 con 0x7f834c0f5930 2021-03-09T09:03:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400d9d0 con 0x7f834c0f5930 2021-03-09T09:03:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:14.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404bbb0 con 0x7f834c0f5930 2021-03-09T09:03:14.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:03:14 smithi146 bash[13136]: debug 2021-03-09T09:03:14.382+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 180 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:03:19.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338011d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:03:19.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338011d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:19.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338011d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f8344020d00 tx=0x7f83440502f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:19.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338011d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f8344020d00 tx=0x7f83440502f0).stop 2021-03-09T09:03:19.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338011d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:19.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338011d40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:19.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:19.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:19.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:03:19.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:03:19.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:19.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:19.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:19.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833804f350 con 0x7f834c0f4690 2021-03-09T09:03:19.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806b3a0 con 0x7f834c0f6280 2021-03-09T09:03:19.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802eba0 con 0x7f8338062ca0 2021-03-09T09:03:19.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:19.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.384+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:19.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:19.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.384+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:19.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:19.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.384+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ecf0 con 0x7f834c0f6280 2021-03-09T09:03:19.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801b660 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f833c006bd0 tx=0x7f833c072000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:19.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068040 con 0x7f834c0f6280 2021-03-09T09:03:19.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00c750 con 0x7f834c0f6280 2021-03-09T09:03:19.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:19.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c020a60 con 0x7f834c0f6280 2021-03-09T09:03:19.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:03:19 smithi146 bash[13136]: debug 2021-03-09T09:03:19.382+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 205 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:03:24.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:03:24.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:24.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801b660 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f833c006bd0 tx=0x7f833c072000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:24.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801b660 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f833c006bd0 tx=0x7f833c072000).stop 2021-03-09T09:03:24.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801b660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:24.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801b660 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:24.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:24.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338012480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:24.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:03:24.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:24.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:24.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338012480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:24.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:03:24.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:24.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833804f350 con 0x7f834c0f4690 2021-03-09T09:03:24.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069d00 con 0x7f834c0f5930 2021-03-09T09:03:24.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2b0 con 0x7f8338062ca0 2021-03-09T09:03:24.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:24.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:24.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:24.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:24.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380677b0 con 0x7f834c0f5930 2021-03-09T09:03:24.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338012480 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f834400df80 tx=0x7f83440502f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:24.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344063fb0 con 0x7f834c0f5930 2021-03-09T09:03:24.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400d9d0 con 0x7f834c0f5930 2021-03-09T09:03:24.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:24.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404bbb0 con 0x7f834c0f5930 2021-03-09T09:03:24.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:03:24 smithi146 bash[13136]: debug 2021-03-09T09:03:24.381+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 229 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:03:29.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338012480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:03:29.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338012480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:29.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338012480 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f834400df80 tx=0x7f83440502f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:29.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338012480 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f834400df80 tx=0x7f83440502f0).stop 2021-03-09T09:03:29.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338012480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:29.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338012480 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:29.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:29.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:29.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:29.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067800 con 0x7f834c0f4690 2021-03-09T09:03:29.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338029000 con 0x7f834c0f6280 2021-03-09T09:03:29.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380168c0 con 0x7f8338062ca0 2021-03-09T09:03:29.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:03:29.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:03:29.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:29.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:29.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:29.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:29.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:29.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:29.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:29.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380712c0 con 0x7f834c0f6280 2021-03-09T09:03:29.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801b660 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f8338043200 tx=0x7f83440502f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:29.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344065db0 con 0x7f834c0f6280 2021-03-09T09:03:29.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400d9d0 con 0x7f834c0f6280 2021-03-09T09:03:29.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:29.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834406f480 con 0x7f834c0f6280 2021-03-09T09:03:29.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:03:29 smithi146 bash[13136]: debug 2021-03-09T09:03:29.385+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 253 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:03:34.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.383+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:03:34.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.383+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:34.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.383+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801b660 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f8338043200 tx=0x7f83440502f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:34.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.383+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801b660 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f8338043200 tx=0x7f83440502f0).stop 2021-03-09T09:03:34.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801b660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:34.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.383+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801b660 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:34.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.383+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:34.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.383+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338012d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:34.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.383+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:34.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016910 con 0x7f834c0f4690 2021-03-09T09:03:34.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028fd0 con 0x7f834c0f5930 2021-03-09T09:03:34.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833804f350 con 0x7f8338062ca0 2021-03-09T09:03:34.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.383+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:03:34.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.383+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:34.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.387+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338012d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:34.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.387+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:03:34.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.387+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:34.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.387+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:34.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.387+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:34.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.387+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:34.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.387+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:34.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.387+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007500 con 0x7f834c0f5930 2021-03-09T09:03:34.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.387+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338012d70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f83440502f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:34.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.391+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344014a10 con 0x7f834c0f5930 2021-03-09T09:03:34.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.391+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344014830 con 0x7f834c0f5930 2021-03-09T09:03:34.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:34.395+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023080 con 0x7f834c0f5930 2021-03-09T09:03:34.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:03:34 smithi146 bash[13136]: debug 2021-03-09T09:03:34.385+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 278 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:03:39.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.383+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338012d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:03:39.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.383+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338012d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:39.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.383+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338012d70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f83440502f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:39.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.383+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338012d70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f83440502f0).stop 2021-03-09T09:03:39.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338012d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:39.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.383+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338012d70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:39.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:39.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:39.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:39.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833803c260 con 0x7f834c0f4690 2021-03-09T09:03:39.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338029000 con 0x7f834c0f6280 2021-03-09T09:03:39.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380168d0 con 0x7f8338062ca0 2021-03-09T09:03:39.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:03:39.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:39.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:03:39.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:39.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:39.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:39.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:39.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:39.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:39.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833804f350 con 0x7f834c0f6280 2021-03-09T09:03:39.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801c0f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f833c022510 tx=0x7f833c006a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:39.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068030 con 0x7f834c0f6280 2021-03-09T09:03:39.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04e940 con 0x7f834c0f6280 2021-03-09T09:03:39.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:39.391+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f834c0f6280 2021-03-09T09:03:39.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:03:39 smithi146 bash[13136]: debug 2021-03-09T09:03:39.385+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 302 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:03:44.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:03:44.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:44.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801c0f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f833c022510 tx=0x7f833c006a30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:44.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801c0f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f833c022510 tx=0x7f833c006a30).stop 2021-03-09T09:03:44.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801c0f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:44.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801c0f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:44.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:44.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:03:44.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:44.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:44.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338013960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:44.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833803c260 con 0x7f834c0f4690 2021-03-09T09:03:44.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338049db0 con 0x7f834c0f5930 2021-03-09T09:03:44.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e810 con 0x7f8338062ca0 2021-03-09T09:03:44.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:03:44.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:44.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.387+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338013960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:44.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.391+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:44.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.391+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:44.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.391+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:44.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.391+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:44.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.391+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007500 con 0x7f834c0f5930 2021-03-09T09:03:44.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.391+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338013960 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f833c0716d0 tx=0x7f833c006a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:44.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.391+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05c020 con 0x7f834c0f5930 2021-03-09T09:03:44.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.391+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05c020 con 0x7f834c0f5930 2021-03-09T09:03:44.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:44.395+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c071c00 con 0x7f834c0f5930 2021-03-09T09:03:44.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:03:44 smithi146 bash[13136]: debug 2021-03-09T09:03:44.385+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 327 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:03:49.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338013960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:03:49.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338013960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:49.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338013960 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f833c0716d0 tx=0x7f833c006a30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:49.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338013960 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f833c0716d0 tx=0x7f833c006a30).stop 2021-03-09T09:03:49.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338013960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:49.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338013960 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:49.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:49.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833803c260 con 0x7f834c0f4690 2021-03-09T09:03:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f834c0f6280 2021-03-09T09:03:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338049db0 con 0x7f8338062ca0 2021-03-09T09:03:49.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:03:49.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:49.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:03:49.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:49.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.387+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:49.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.391+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:49.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.391+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:49.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.391+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:49.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.391+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:49.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.391+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022220 con 0x7f834c0f6280 2021-03-09T09:03:49.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.391+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801db00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c006a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:49.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.391+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c02ce40 con 0x7f834c0f6280 2021-03-09T09:03:49.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.395+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c029300 con 0x7f834c0f6280 2021-03-09T09:03:49.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:49.399+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c029020 con 0x7f834c0f6280 2021-03-09T09:03:49.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:03:49 smithi146 bash[13136]: debug 2021-03-09T09:03:49.385+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 351 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:03:54.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.387+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:03:54.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.387+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:54.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.387+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801db00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c006a30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:54.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.387+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801db00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c006a30).stop 2021-03-09T09:03:54.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801db00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:54.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801db00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:54.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:54.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:54.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.387+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:03:54.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.387+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:54.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338014220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:54.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.387+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:03:54.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.387+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:54.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.391+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338049db0 con 0x7f834c0f4690 2021-03-09T09:03:54.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.391+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338049570 con 0x7f834c0f5930 2021-03-09T09:03:54.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.391+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bdb0 con 0x7f8338062ca0 2021-03-09T09:03:54.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.391+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338014220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:54.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.391+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:54.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.391+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:54.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.391+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:54.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.391+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:54.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.391+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f834c0f5930 2021-03-09T09:03:54.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.391+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338014220 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f833802ba20 tx=0x7f833c021f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:54.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.391+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0167d0 con 0x7f834c0f5930 2021-03-09T09:03:54.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.391+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c016180 con 0x7f834c0f5930 2021-03-09T09:03:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:54.395+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05eb10 con 0x7f834c0f5930 2021-03-09T09:03:54.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:03:54 smithi146 bash[13136]: debug 2021-03-09T09:03:54.385+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 376 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:03:59.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.387+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338014220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:03:59.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.387+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338014220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:59.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.387+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338014220 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f833802ba20 tx=0x7f833c021f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:59.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.387+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338014220 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f833802ba20 tx=0x7f833c021f50).stop 2021-03-09T09:03:59.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338014220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:59.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338014220 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:59.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:59.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.387+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:03:59.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.387+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:59.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:59.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338006790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:59.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338049db0 con 0x7f834c0f4690 2021-03-09T09:03:59.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338031150 con 0x7f834c0f6280 2021-03-09T09:03:59.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338049570 con 0x7f8338062ca0 2021-03-09T09:03:59.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.387+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:03:59.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.387+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:59.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.391+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338006790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:59.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.391+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:59.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.391+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:59.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.395+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:59.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.395+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:59.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.395+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833804f350 con 0x7f834c0f6280 2021-03-09T09:03:59.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.395+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338006790 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f833c019010 tx=0x7f833c021f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:59.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.395+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f834c0f6280 2021-03-09T09:03:59.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.399+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c016180 con 0x7f834c0f6280 2021-03-09T09:03:59.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:03:59.399+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c066140 con 0x7f834c0f6280 2021-03-09T09:03:59.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:03:59 smithi146 bash[13136]: debug 2021-03-09T09:03:59.385+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 400 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:04:04.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.387+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338006790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:04:04.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.387+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338006790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:04.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.387+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338006790 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f833c019010 tx=0x7f833c021f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:04.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.387+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338006790 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f833c019010 tx=0x7f833c021f50).stop 2021-03-09T09:04:04.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338006790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:04.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338006790 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:04.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338079c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:04.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:04.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.387+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:04.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338049570 con 0x7f834c0f4690 2021-03-09T09:04:04.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bac0 con 0x7f834c0f5930 2021-03-09T09:04:04.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.387+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022de0 con 0x7f8338062ca0 2021-03-09T09:04:04.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.391+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:04:04.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.391+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:04.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.391+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338079c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:04.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.391+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:04:04.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.391+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:04.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.391+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:04.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.391+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:04.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.391+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:04.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.391+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:04.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.391+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380677b0 con 0x7f834c0f5930 2021-03-09T09:04:04.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.391+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338079c60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833400b6c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:04.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.391+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f834c0f5930 2021-03-09T09:04:04.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.391+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334004030 con 0x7f834c0f5930 2021-03-09T09:04:04.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:04.395+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334007a40 con 0x7f834c0f5930 2021-03-09T09:04:04.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:04:04 smithi146 bash[13136]: debug 2021-03-09T09:04:04.389+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 425 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:04:09.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338079c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:04:09.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338079c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:09.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338079c60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833400b6c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:09.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338079c60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833400b6c0).stop 2021-03-09T09:04:09.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338079c60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:09.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338079c60 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:09.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:09.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:04:09.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:09.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:09.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380058b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:09.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338031150 con 0x7f834c0f4690 2021-03-09T09:04:09.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338049db0 con 0x7f834c0f6280 2021-03-09T09:04:09.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338049570 con 0x7f8338062ca0 2021-03-09T09:04:09.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:04:09.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:09.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.390+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380058b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:09.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.394+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:09.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.394+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:09.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.394+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:09.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.394+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:09.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.394+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833803c210 con 0x7f834c0f6280 2021-03-09T09:04:09.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.394+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380058b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f833c009f90 tx=0x7f833c024040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:09.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.394+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068030 con 0x7f834c0f6280 2021-03-09T09:04:09.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c055730 con 0x7f834c0f6280 2021-03-09T09:04:09.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:09.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c015370 con 0x7f834c0f6280 2021-03-09T09:04:09.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:04:09 smithi146 bash[13136]: debug 2021-03-09T09:04:09.389+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 449 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:04:14.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380058b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:04:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380058b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:14.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380058b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f833c009f90 tx=0x7f833c024040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380058b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f833c009f90 tx=0x7f833c024040).stop 2021-03-09T09:04:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380058b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380058b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:14.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:04:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833800b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:14.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016910 con 0x7f834c0f4690 2021-03-09T09:04:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338049570 con 0x7f834c0f5930 2021-03-09T09:04:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338049db0 con 0x7f8338062ca0 2021-03-09T09:04:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:04:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833800b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:14.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.390+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.394+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.394+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:14.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.394+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338044090 con 0x7f834c0f5930 2021-03-09T09:04:14.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.394+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833800b1c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f8338015440 tx=0x7f83440502f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:14.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.394+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344065ce0 con 0x7f834c0f5930 2021-03-09T09:04:14.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.394+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344054610 con 0x7f834c0f5930 2021-03-09T09:04:14.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:14.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344054910 con 0x7f834c0f5930 2021-03-09T09:04:14.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:04:14 smithi146 bash[13136]: debug 2021-03-09T09:04:14.389+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 474 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:04:19.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833800b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:04:19.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833800b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:19.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833800b1c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f8338015440 tx=0x7f83440502f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:19.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833800b1c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f8338015440 tx=0x7f83440502f0).stop 2021-03-09T09:04:19.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833800b1c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:19.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833800b1c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:19.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:19.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:04:19.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:19.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:19.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338079c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:19.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338031150 con 0x7f834c0f4690 2021-03-09T09:04:19.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338049db0 con 0x7f834c0f6280 2021-03-09T09:04:19.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071310 con 0x7f8338062ca0 2021-03-09T09:04:19.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:04:19.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:19.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.390+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338079c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:19.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.394+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:19.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.394+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:19.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.394+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:19.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.394+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:19.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.394+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833803c210 con 0x7f834c0f6280 2021-03-09T09:04:19.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.394+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338079c60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f8338016c20 tx=0x7f833c024040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:19.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f834c0f6280 2021-03-09T09:04:19.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c016180 con 0x7f834c0f6280 2021-03-09T09:04:19.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:19.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f834c0f6280 2021-03-09T09:04:19.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:04:19 smithi146 bash[13136]: debug 2021-03-09T09:04:19.389+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 498 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:04:24.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.390+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338079c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:04:24.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.390+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338079c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:24.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.390+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338079c60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f8338016c20 tx=0x7f833c024040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:24.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.390+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338079c60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f8338016c20 tx=0x7f833c024040).stop 2021-03-09T09:04:24.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338079c60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:24.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338079c60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:24.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:24.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.390+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:04:24.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.390+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:24.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833807a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:24.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:24.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338049570 con 0x7f834c0f4690 2021-03-09T09:04:24.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338031150 con 0x7f834c0f5930 2021-03-09T09:04:24.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071310 con 0x7f8338062ca0 2021-03-09T09:04:24.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.390+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833807a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:24.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.394+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:04:24.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.394+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:24.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.394+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:24.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.394+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:24.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.394+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:24.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.394+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:24.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.394+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380168c0 con 0x7f834c0f5930 2021-03-09T09:04:24.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.394+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833807a150 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f834405cd90 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:24.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.394+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344062070 con 0x7f834c0f5930 2021-03-09T09:04:24.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.394+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344016e00 con 0x7f834c0f5930 2021-03-09T09:04:24.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:24.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023080 con 0x7f834c0f5930 2021-03-09T09:04:24.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:04:24 smithi146 bash[13136]: debug 2021-03-09T09:04:24.393+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 523 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:04:29.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833807a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:04:29.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833807a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:29.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833807a150 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f834405cd90 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:29.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833807a150 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f834405cd90 tx=0x7f8344003f90).stop 2021-03-09T09:04:29.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833807a150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:29.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833807a150 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:29.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:29.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:04:29.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:29.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:29.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:29.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b440 con 0x7f834c0f4690 2021-03-09T09:04:29.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338037480 con 0x7f834c0f6280 2021-03-09T09:04:29.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338031150 con 0x7f8338062ca0 2021-03-09T09:04:29.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:04:29.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:29.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.390+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:29.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.394+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:29.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.394+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:29.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.394+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:29.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.394+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:29.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.394+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a740 con 0x7f834c0f6280 2021-03-09T09:04:29.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.394+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025500 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c006a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:29.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.394+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022380 con 0x7f834c0f6280 2021-03-09T09:04:29.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05ea10 con 0x7f834c0f6280 2021-03-09T09:04:29.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:29.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c078ca0 con 0x7f834c0f6280 2021-03-09T09:04:29.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:04:29 smithi146 bash[13136]: debug 2021-03-09T09:04:29.392+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 547 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:04:34.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.390+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338025500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:04:34.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.390+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338025500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:34.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.390+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025500 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c006a30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:34.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.390+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025500 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c006a30).stop 2021-03-09T09:04:34.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338025500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:34.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025500 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:34.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:34.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338020010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:34.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:34.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e540 con 0x7f834c0f4690 2021-03-09T09:04:34.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f834c0f5930 2021-03-09T09:04:34.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338031150 con 0x7f8338062ca0 2021-03-09T09:04:34.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.394+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338020010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:34.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.394+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:04:34.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.394+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:34.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.394+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:04:34.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.394+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:34.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.394+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:34.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.394+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:34.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.394+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:34.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.394+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:34.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.394+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338047b90 con 0x7f834c0f5930 2021-03-09T09:04:34.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.394+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338020010 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f8338037190 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:34.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.394+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401e730 con 0x7f834c0f5930 2021-03-09T09:04:34.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.394+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834401e910 con 0x7f834c0f5930 2021-03-09T09:04:34.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:34.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023070 con 0x7f834c0f5930 2021-03-09T09:04:34.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:04:34 smithi146 bash[13136]: debug 2021-03-09T09:04:34.392+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 571 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:04:39.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338020010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:04:39.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338020010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:39.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338020010 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f8338037190 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:39.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338020010 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f8338037190 tx=0x7f8344003f90).stop 2021-03-09T09:04:39.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338020010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:39.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338020010 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:39.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:39.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:04:39.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:39.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:39.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:39.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380712d0 con 0x7f834c0f4690 2021-03-09T09:04:39.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338037480 con 0x7f834c0f6280 2021-03-09T09:04:39.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338031150 con 0x7f8338062ca0 2021-03-09T09:04:39.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:04:39.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:39.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:39.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:39.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:39.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:39.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:39.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.393+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338044090 con 0x7f834c0f6280 2021-03-09T09:04:39.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.397+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025500 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f8338012130 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:39.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404eaa0 con 0x7f834c0f6280 2021-03-09T09:04:39.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404e8c0 con 0x7f834c0f6280 2021-03-09T09:04:39.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:39.401+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834401e2f0 con 0x7f834c0f6280 2021-03-09T09:04:39.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:04:39 smithi146 bash[13136]: debug 2021-03-09T09:04:39.388+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 596 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:04:44.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338025500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:04:44.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338025500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:44.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025500 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f8338012130 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:44.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025500 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f8338012130 tx=0x7f8344003f90).stop 2021-03-09T09:04:44.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338025500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:44.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025500 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:44.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833800ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:44.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:44.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:44.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e540 con 0x7f834c0f4690 2021-03-09T09:04:44.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338031150 con 0x7f834c0f5930 2021-03-09T09:04:44.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338037480 con 0x7f8338062ca0 2021-03-09T09:04:44.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:04:44.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:44.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833800ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:44.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:04:44.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:44.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.393+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833803c210 con 0x7f834c0f5930 2021-03-09T09:04:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.397+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833800ad30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f8334009630 tx=0x7f833400b6c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:44.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f834c0f5930 2021-03-09T09:04:44.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007bf0 con 0x7f834c0f5930 2021-03-09T09:04:44.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:44.401+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402abe0 con 0x7f834c0f5930 2021-03-09T09:04:44.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:04:44 smithi146 bash[13136]: debug 2021-03-09T09:04:44.392+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 620 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:04:49.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833800ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:04:49.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833800ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:49.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833800ad30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f8334009630 tx=0x7f833400b6c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:49.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833800ad30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f8334009630 tx=0x7f833400b6c0).stop 2021-03-09T09:04:49.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833800ad30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:49.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833800ad30 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:49.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:49.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:04:49.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:49.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:49.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:49.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338037480 con 0x7f834c0f4690 2021-03-09T09:04:49.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062920 con 0x7f834c0f6280 2021-03-09T09:04:49.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8338062ca0 2021-03-09T09:04:49.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:04:49.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:49.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:49.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:49.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:49.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:49.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:49.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.393+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e640 con 0x7f834c0f6280 2021-03-09T09:04:49.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.397+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f833c060770 tx=0x7f833c006a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:49.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c02c1d0 con 0x7f834c0f6280 2021-03-09T09:04:49.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c02c1d0 con 0x7f834c0f6280 2021-03-09T09:04:49.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:49.401+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f834c0f6280 2021-03-09T09:04:49.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:04:49 smithi146 bash[13136]: debug 2021-03-09T09:04:49.392+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 645 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:04:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338018e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:04:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338018e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f833c060770 tx=0x7f833c006a30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f833c060770 tx=0x7f833c006a30).stop 2021-03-09T09:04:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338018e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:54.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:54.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:54.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380147f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:54.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012790 con 0x7f834c0f4690 2021-03-09T09:04:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338031150 con 0x7f834c0f5930 2021-03-09T09:04:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338037480 con 0x7f8338062ca0 2021-03-09T09:04:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:04:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:04:54.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:54.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.393+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380147f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:54.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.397+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:54.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.397+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:54.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.397+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:54.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.397+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:54.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.397+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338044090 con 0x7f834c0f5930 2021-03-09T09:04:54.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.401+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380147f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f833c007010 tx=0x7f833c006a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:54.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.401+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f834c0f5930 2021-03-09T09:04:54.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.401+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0135c0 con 0x7f834c0f5930 2021-03-09T09:04:54.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:54.405+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c013010 con 0x7f834c0f5930 2021-03-09T09:04:54.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:04:54 smithi146 bash[13136]: debug 2021-03-09T09:04:54.396+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 669 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:04:59.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f83380147f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:04:59.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f83380147f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:59.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380147f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f833c007010 tx=0x7f833c006a30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:59.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380147f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f833c007010 tx=0x7f833c006a30).stop 2021-03-09T09:04:59.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f83380147f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:59.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380147f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:59.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:59.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:04:59.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:59.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:59.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:59.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380258b0 con 0x7f834c0f4690 2021-03-09T09:04:59.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338037480 con 0x7f834c0f6280 2021-03-09T09:04:59.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806b8a0 con 0x7f8338062ca0 2021-03-09T09:04:59.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:04:59.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:59.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.393+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:59.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.397+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:59.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.397+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:59.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.397+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:59.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.397+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:59.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.397+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380259f0 con 0x7f834c0f6280 2021-03-09T09:04:59.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.397+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f834402ab70 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:59.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029040 con 0x7f834c0f6280 2021-03-09T09:04:59.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.401+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402a600 con 0x7f834c0f6280 2021-03-09T09:04:59.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:04:59.401+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023040 con 0x7f834c0f6280 2021-03-09T09:04:59.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:04:59 smithi146 bash[13136]: debug 2021-03-09T09:04:59.396+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 694 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:05:04.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338018e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:05:04.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338018e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:04.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f834402ab70 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:04.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f834402ab70 tx=0x7f8344003f90).stop 2021-03-09T09:05:04.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338018e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:04.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:04.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338066960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:04.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:04.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012790 con 0x7f834c0f4690 2021-03-09T09:05:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338031150 con 0x7f834c0f5930 2021-03-09T09:05:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338037480 con 0x7f8338062ca0 2021-03-09T09:05:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:05:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:04.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:05:04.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:04.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338066960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:04.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:04.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:04.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:04.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:04.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833803c210 con 0x7f834c0f5930 2021-03-09T09:05:04.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338066960 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f8334009b40 tx=0x7f833400b6c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:04.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f834c0f5930 2021-03-09T09:05:04.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334004040 con 0x7f834c0f5930 2021-03-09T09:05:04.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:04.401+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334028640 con 0x7f834c0f5930 2021-03-09T09:05:04.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:05:04 smithi146 bash[13136]: debug 2021-03-09T09:05:04.396+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 717 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:05:09.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.393+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338066960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:05:09.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.393+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338066960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:09.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.393+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338066960 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f8334009b40 tx=0x7f833400b6c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:09.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.393+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338066960 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f8334009b40 tx=0x7f833400b6c0).stop 2021-03-09T09:05:09.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.393+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338066960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:09.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.393+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338066960 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:09.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.397+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:09.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.397+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:05:09.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.397+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:09.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.397+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:09.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.397+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:09.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338037480 con 0x7f834c0f4690 2021-03-09T09:05:09.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024b80 con 0x7f834c0f6280 2021-03-09T09:05:09.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069f30 con 0x7f8338062ca0 2021-03-09T09:05:09.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.397+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:05:09.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.397+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:09.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.397+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:09.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.401+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:09.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.401+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:09.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.401+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:09.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.401+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:09.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.401+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338044090 con 0x7f834c0f6280 2021-03-09T09:05:09.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.401+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344070040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:09.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.401+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344071ed0 con 0x7f834c0f6280 2021-03-09T09:05:09.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.401+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402a600 con 0x7f834c0f6280 2021-03-09T09:05:09.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:09.405+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344054190 con 0x7f834c0f6280 2021-03-09T09:05:09.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:05:09 smithi146 bash[13136]: debug 2021-03-09T09:05:09.396+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 743 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:05:14.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338018e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:05:14.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338018e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:14.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344070040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:14.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344070040).stop 2021-03-09T09:05:14.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338018e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:14.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338018e10 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338070a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:14.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:05:14.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:05:14.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:14.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:14.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338031150 con 0x7f834c0f4690 2021-03-09T09:05:14.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069f30 con 0x7f834c0f5930 2021-03-09T09:05:14.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338070a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:14.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fbb0 con 0x7f8338062ca0 2021-03-09T09:05:14.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.401+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:14.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.401+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:14.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.401+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:14.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.401+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:14.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.401+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833803c210 con 0x7f834c0f5930 2021-03-09T09:05:14.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.401+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338070a90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f833402bab0 tx=0x7f833400b6c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:14.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.409+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833402b5a0 con 0x7f834c0f5930 2021-03-09T09:05:14.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.409+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334004040 con 0x7f834c0f5930 2021-03-09T09:05:14.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:14.409+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334068e70 con 0x7f834c0f5930 2021-03-09T09:05:14.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:05:14 smithi146 bash[13136]: debug 2021-03-09T09:05:14.400+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 766 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:05:19.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338070a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:05:19.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338070a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:19.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338070a90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f833402bab0 tx=0x7f833400b6c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:19.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338070a90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f833402bab0 tx=0x7f833400b6c0).stop 2021-03-09T09:05:19.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338070a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:19.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338070a90 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:19.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:19.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:19.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338017b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:19.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022de0 con 0x7f834c0f4690 2021-03-09T09:05:19.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338031150 con 0x7f834c0f6280 2021-03-09T09:05:19.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800be30 con 0x7f8338062ca0 2021-03-09T09:05:19.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:05:19.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:19.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:05:19.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:19.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.397+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338017b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:19.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.401+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:19.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.401+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:19.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.401+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:19.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.401+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:19.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.401+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833803c210 con 0x7f834c0f6280 2021-03-09T09:05:19.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.401+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338017b90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f833c013960 tx=0x7f833c004f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:19.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.401+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c077550 con 0x7f834c0f6280 2021-03-09T09:05:19.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.401+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f834c0f6280 2021-03-09T09:05:19.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:19.405+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04de90 con 0x7f834c0f6280 2021-03-09T09:05:19.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:05:19 smithi146 bash[13136]: debug 2021-03-09T09:05:19.400+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 792 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:05:24.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.396+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338017b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:05:24.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.396+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338017b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:24.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.396+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338017b90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f833c013960 tx=0x7f833c004f60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:24.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.396+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338017b90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f833c013960 tx=0x7f833c004f60).stop 2021-03-09T09:05:24.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338017b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:24.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338017b90 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:24.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:24.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338005870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:24.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:24.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069f30 con 0x7f834c0f4690 2021-03-09T09:05:24.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d540 con 0x7f834c0f5930 2021-03-09T09:05:24.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064c50 con 0x7f8338062ca0 2021-03-09T09:05:24.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.396+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:05:24.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.396+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:24.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.396+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:24.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:05:24.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:24.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.400+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:24.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.400+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:24.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.400+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:24.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.400+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:24.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.400+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014e70 con 0x7f834c0f5930 2021-03-09T09:05:24.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.400+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338005870 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344070040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:24.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029040 con 0x7f834c0f5930 2021-03-09T09:05:24.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834401f2f0 con 0x7f834c0f5930 2021-03-09T09:05:24.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:24.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023030 con 0x7f834c0f5930 2021-03-09T09:05:24.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:05:24 smithi146 bash[13136]: debug 2021-03-09T09:05:24.400+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 815 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:05:29.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.396+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338005870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:05:29.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.396+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338005870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:29.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.396+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338005870 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344070040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:29.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.396+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338005870 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344070040).stop 2021-03-09T09:05:29.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338005870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:29.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338005870 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:29.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:29.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:29.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:29.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014180 con 0x7f834c0f4690 2021-03-09T09:05:29.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a790 con 0x7f834c0f6280 2021-03-09T09:05:29.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380114e0 con 0x7f8338062ca0 2021-03-09T09:05:29.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.396+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:05:29.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.396+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:29.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.400+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:05:29.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.400+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:29.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:29.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:29.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:29.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:29.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:29.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338032450 con 0x7f834c0f6280 2021-03-09T09:05:29.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f833c067700 tx=0x7f833c023000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:29.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f834c0f6280 2021-03-09T09:05:29.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f834c0f6280 2021-03-09T09:05:29.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:29.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0041c0 con 0x7f834c0f6280 2021-03-09T09:05:29.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:05:29 smithi146 bash[13136]: debug 2021-03-09T09:05:29.400+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 841 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:05:34.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.396+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:05:34.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.396+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:34.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.396+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f833c067700 tx=0x7f833c023000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:34.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.396+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f833c067700 tx=0x7f833c023000).stop 2021-03-09T09:05:34.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806c690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:34.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:34.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:34.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:34.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338006200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:34.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064c50 con 0x7f834c0f4690 2021-03-09T09:05:34.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380114e0 con 0x7f834c0f5930 2021-03-09T09:05:34.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061fd0 con 0x7f8338062ca0 2021-03-09T09:05:34.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.400+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:05:34.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.400+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:34.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.400+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:05:34.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.400+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:34.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338006200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:34.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:34.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:34.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:34.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:34.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ea20 con 0x7f834c0f5930 2021-03-09T09:05:34.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338006200 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f8338025120 tx=0x7f833c023000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:34.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c052e20 con 0x7f834c0f5930 2021-03-09T09:05:34.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00c5d0 con 0x7f834c0f5930 2021-03-09T09:05:34.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:34.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00d280 con 0x7f834c0f5930 2021-03-09T09:05:34.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:05:34 smithi146 bash[13136]: debug 2021-03-09T09:05:34.400+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 864 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:05:39.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338006200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:05:39.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338006200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:39.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338006200 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f8338025120 tx=0x7f833c023000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:39.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338006200 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f8338025120 tx=0x7f833c023000).stop 2021-03-09T09:05:39.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338006200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:39.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338006200 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:39.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:39.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:39.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:39.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011330 con 0x7f834c0f4690 2021-03-09T09:05:39.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fd50 con 0x7f834c0f6280 2021-03-09T09:05:39.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380114e0 con 0x7f8338062ca0 2021-03-09T09:05:39.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:05:39.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:39.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:39.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:39.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:39.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-03-09T09:05:39.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:05:39.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.400+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:39.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.404+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:39.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.404+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a5e0 con 0x7f834c0f6280 2021-03-09T09:05:39.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.404+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806cf40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f833402ecd0 tx=0x7f833400b6c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:39.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833402b7b0 con 0x7f834c0f6280 2021-03-09T09:05:39.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334004040 con 0x7f834c0f6280 2021-03-09T09:05:39.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:39.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334007a50 con 0x7f834c0f6280 2021-03-09T09:05:39.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:05:39 smithi146 bash[13136]: debug 2021-03-09T09:05:39.404+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 889 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:05:44.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:05:44.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:44.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806cf40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f833402ecd0 tx=0x7f833400b6c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:44.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806cf40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f833402ecd0 tx=0x7f833400b6c0).stop 2021-03-09T09:05:44.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806cf40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:44.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806cf40 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:44.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:44.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:44.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:44.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d90 con 0x7f834c0f4690 2021-03-09T09:05:44.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014180 con 0x7f834c0f5930 2021-03-09T09:05:44.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d0c0 con 0x7f8338062ca0 2021-03-09T09:05:44.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:05:44.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:05:44.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:44.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:44.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:44.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.400+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022d90 con 0x7f834c0f5930 2021-03-09T09:05:44.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.404+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f83380146c0 tx=0x7f8344070040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:44.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f834c0f5930 2021-03-09T09:05:44.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440484d0 con 0x7f834c0f5930 2021-03-09T09:05:44.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:44.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023070 con 0x7f834c0f5930 2021-03-09T09:05:44.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:05:44 smithi146 bash[13136]: debug 2021-03-09T09:05:44.404+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 912 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:05:49.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338027b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:05:49.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338027b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:49.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f83380146c0 tx=0x7f8344070040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:49.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f83380146c0 tx=0x7f8344070040).stop 2021-03-09T09:05:49.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338027b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:49.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b30 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:49.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:49.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:49.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:49.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eac0 con 0x7f834c0f4690 2021-03-09T09:05:49.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d90 con 0x7f834c0f6280 2021-03-09T09:05:49.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022de0 con 0x7f8338062ca0 2021-03-09T09:05:49.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:05:49.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:49.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:05:49.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:49.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.400+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:49.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.404+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:49.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.404+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:49.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.404+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:49.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.404+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:49.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.404+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338032450 con 0x7f834c0f6280 2021-03-09T09:05:49.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.404+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f83340139e0 tx=0x7f833400b6c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:49.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334046050 con 0x7f834c0f6280 2021-03-09T09:05:49.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334004030 con 0x7f834c0f6280 2021-03-09T09:05:49.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:49.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334007970 con 0x7f834c0f6280 2021-03-09T09:05:49.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:05:49 smithi146 bash[13136]: debug 2021-03-09T09:05:49.404+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 937 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:05:54.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:05:54.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:54.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f83340139e0 tx=0x7f833400b6c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:54.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f83340139e0 tx=0x7f833400b6c0).stop 2021-03-09T09:05:54.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338067ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:54.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:54.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:54.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:54.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380683d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:54.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338033d70 con 0x7f834c0f4690 2021-03-09T09:05:54.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d90 con 0x7f834c0f5930 2021-03-09T09:05:54.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338029030 con 0x7f8338062ca0 2021-03-09T09:05:54.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:05:54.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:54.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:05:54.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:54.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380683d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:54.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:54.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:54.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:54.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:54.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338032450 con 0x7f834c0f5930 2021-03-09T09:05:54.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380683d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f833c05fbf0 tx=0x7f833c023000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:54.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05fa30 con 0x7f834c0f5930 2021-03-09T09:05:54.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05fc60 con 0x7f834c0f5930 2021-03-09T09:05:54.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:54.412+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c053980 con 0x7f834c0f5930 2021-03-09T09:05:54.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:05:54 smithi146 bash[13136]: debug 2021-03-09T09:05:54.403+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 961 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:05:59.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f83380683d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:05:59.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f83380683d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:59.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380683d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f833c05fbf0 tx=0x7f833c023000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:59.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380683d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f833c05fbf0 tx=0x7f833c023000).stop 2021-03-09T09:05:59.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f83380683d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:59.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380683d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:59.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:59.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338064570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:59.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:59.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014180 con 0x7f834c0f4690 2021-03-09T09:05:59.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eac0 con 0x7f834c0f6280 2021-03-09T09:05:59.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022de0 con 0x7f8338062ca0 2021-03-09T09:05:59.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:05:59.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:59.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:05:59.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:59.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338064570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:59.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:59.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:59.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:59.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:59.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064a60 con 0x7f834c0f6280 2021-03-09T09:05:59.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338064570 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f833806c4d0 tx=0x7f8344070040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:59.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344048050 con 0x7f834c0f6280 2021-03-09T09:05:59.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834406bd50 con 0x7f834c0f6280 2021-03-09T09:05:59.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:05:59.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400b610 con 0x7f834c0f6280 2021-03-09T09:05:59.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:05:59 smithi146 bash[13136]: debug 2021-03-09T09:05:59.407+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 986 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:06:04.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338064570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:06:04.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338064570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:04.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338064570 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f833806c4d0 tx=0x7f8344070040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:04.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338064570 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f833806c4d0 tx=0x7f8344070040).stop 2021-03-09T09:06:04.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338064570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:04.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338064570 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:04.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:04.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:06:04.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:04.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:04.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338068c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:04.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bc70 con 0x7f834c0f4690 2021-03-09T09:06:04.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338033d70 con 0x7f834c0f5930 2021-03-09T09:06:04.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801bf30 con 0x7f8338062ca0 2021-03-09T09:06:04.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:06:04.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:04.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338068c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:04.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:04.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028fe0 con 0x7f834c0f5930 2021-03-09T09:06:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338068c70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c023000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04e020 con 0x7f834c0f5930 2021-03-09T09:06:04.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c019210 con 0x7f834c0f5930 2021-03-09T09:06:04.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:04.412+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c020a60 con 0x7f834c0f5930 2021-03-09T09:06:04.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:06:04 smithi146 bash[13136]: debug 2021-03-09T09:06:04.407+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1010 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:06:09.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338068c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:06:09.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338068c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:09.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338068c70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c023000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:09.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338068c70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c023000).stop 2021-03-09T09:06:09.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338068c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:09.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338068c70 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:09.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:09.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:06:09.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:09.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:09.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380233b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:09.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b3d0 con 0x7f834c0f4690 2021-03-09T09:06:09.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022de0 con 0x7f834c0f6280 2021-03-09T09:06:09.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ab0 con 0x7f8338062ca0 2021-03-09T09:06:09.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:06:09.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:09.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380233b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:09.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:09.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:09.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:09.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:09.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062960 con 0x7f834c0f6280 2021-03-09T09:06:09.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.412+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380233b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f833c012c60 tx=0x7f833c023000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:09.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.412+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0157c0 con 0x7f834c0f6280 2021-03-09T09:06:09.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.412+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0157c0 con 0x7f834c0f6280 2021-03-09T09:06:09.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:09.416+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c053be0 con 0x7f834c0f6280 2021-03-09T09:06:09.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:06:09 smithi146 bash[13136]: debug 2021-03-09T09:06:09.407+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1035 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:06:14.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380233b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:06:14.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380233b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:14.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380233b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f833c012c60 tx=0x7f833c023000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:14.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380233b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f833c012c60 tx=0x7f833c023000).stop 2021-03-09T09:06:14.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380233b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:14.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380233b0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:14.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:14.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:14.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:14.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e560 con 0x7f834c0f4690 2021-03-09T09:06:14.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801bf30 con 0x7f834c0f5930 2021-03-09T09:06:14.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:06:14.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005420 con 0x7f8338062ca0 2021-03-09T09:06:14.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:14.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:06:14.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:14.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:14.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:14.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:14.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:14.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:14.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338008bd0 con 0x7f834c0f5930 2021-03-09T09:06:14.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f833c035440 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:14.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c015380 con 0x7f834c0f5930 2021-03-09T09:06:14.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.412+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c015380 con 0x7f834c0f5930 2021-03-09T09:06:14.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:14.416+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f834c0f5930 2021-03-09T09:06:14.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:06:14 smithi146 bash[13136]: debug 2021-03-09T09:06:14.407+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1059 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:06:19.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.403+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338027b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:06:19.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.403+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338027b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:19.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.403+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f833c035440 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:19.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.403+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f833c035440 tx=0x7f833c04e870).stop 2021-03-09T09:06:19.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.403+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338027b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:19.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.403+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b30 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:19.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.403+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:19.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.403+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:06:19.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.403+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:19.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.403+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:19.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.403+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:19.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.403+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a630 con 0x7f834c0f4690 2021-03-09T09:06:19.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.403+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021840 con 0x7f834c0f6280 2021-03-09T09:06:19.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.403+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338033d70 con 0x7f8338062ca0 2021-03-09T09:06:19.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.403+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:06:19.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.407+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:19.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.407+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:19.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.407+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:19.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.407+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:19.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.407+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:19.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.407+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:19.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.407+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338025a20 con 0x7f834c0f6280 2021-03-09T09:06:19.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.407+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023320 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f833c052f80 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:19.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c060ba0 con 0x7f834c0f6280 2021-03-09T09:06:19.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c060d00 con 0x7f834c0f6280 2021-03-09T09:06:19.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:19.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c035b50 con 0x7f834c0f6280 2021-03-09T09:06:19.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:06:19 smithi146 bash[13136]: debug 2021-03-09T09:06:19.407+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1083 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:06:24.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.407+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338023320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:06:24.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.407+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338023320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:24.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.407+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023320 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f833c052f80 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:24.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.407+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023320 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f833c052f80 tx=0x7f833c04e870).stop 2021-03-09T09:06:24.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.407+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338023320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:24.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.407+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023320 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:24.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:24.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:24.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:24.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338033d70 con 0x7f834c0f4690 2021-03-09T09:06:24.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380074c0 con 0x7f834c0f5930 2021-03-09T09:06:24.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012c80 con 0x7f8338062ca0 2021-03-09T09:06:24.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.411+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:06:24.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.411+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:24.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:06:24.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:24.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.415+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:24.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.419+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:24.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.419+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:24.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.419+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:24.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.419+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:24.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.419+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800fd50 con 0x7f834c0f5930 2021-03-09T09:06:24.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.419+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b10 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334028ff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:24.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e040 con 0x7f834c0f5930 2021-03-09T09:06:24.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334028ac0 con 0x7f834c0f5930 2021-03-09T09:06:24.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:24.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334007890 con 0x7f834c0f5930 2021-03-09T09:06:24.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:06:24 smithi146 bash[13136]: debug 2021-03-09T09:06:24.411+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1109 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:06:29.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338027b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:06:29.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338027b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:29.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b10 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334028ff0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:29.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b10 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334028ff0).stop 2021-03-09T09:06:29.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338027b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:29.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338027b10 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:29.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:29.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:06:29.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:29.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:29.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:29.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338033d70 con 0x7f834c0f4690 2021-03-09T09:06:29.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338070420 con 0x7f834c0f6280 2021-03-09T09:06:29.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380051a0 con 0x7f8338062ca0 2021-03-09T09:06:29.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:06:29.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:29.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:29.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:29.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:29.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:29.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:29.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b230 con 0x7f834c0f6280 2021-03-09T09:06:29.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023140 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f833c000be0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:29.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c019210 con 0x7f834c0f6280 2021-03-09T09:06:29.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c019210 con 0x7f834c0f6280 2021-03-09T09:06:29.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:29.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00db30 con 0x7f834c0f6280 2021-03-09T09:06:29.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:06:29 smithi146 bash[13136]: debug 2021-03-09T09:06:29.411+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1133 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:06:34.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338023140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:06:34.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338023140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:34.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023140 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f833c000be0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:34.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023140 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f833c000be0 tx=0x7f833c04e870).stop 2021-03-09T09:06:34.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338023140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:34.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023140 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:34.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:34.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:06:34.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:34.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:34.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338023d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:34.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338033d70 con 0x7f834c0f4690 2021-03-09T09:06:34.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d160 con 0x7f834c0f5930 2021-03-09T09:06:34.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a630 con 0x7f8338062ca0 2021-03-09T09:06:34.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:06:34.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:34.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338023d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:34.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:34.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:34.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:34.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:34.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338061f80 con 0x7f834c0f5930 2021-03-09T09:06:34.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338023d20 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:34.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0153b0 con 0x7f834c0f5930 2021-03-09T09:06:34.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c06ab50 con 0x7f834c0f5930 2021-03-09T09:06:34.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:34.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c035610 con 0x7f834c0f5930 2021-03-09T09:06:34.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:06:34 smithi146 bash[13136]: debug 2021-03-09T09:06:34.411+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1159 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:06:39.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338023d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:06:39.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338023d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:39.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338023d20 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:39.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338023d20 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c04e870).stop 2021-03-09T09:06:39.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f8338023d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:39.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f8338023d20 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:39.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:39.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:06:39.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:39.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338013d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:39.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:39.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338033d70 con 0x7f834c0f4690 2021-03-09T09:06:39.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a630 con 0x7f834c0f6280 2021-03-09T09:06:39.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021840 con 0x7f8338062ca0 2021-03-09T09:06:39.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:06:39.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:39.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.411+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338013d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:39.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:39.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:39.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:39.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:39.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380030c0 con 0x7f834c0f6280 2021-03-09T09:06:39.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338013d50 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f8338062510 tx=0x7f8344048960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:39.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401eca0 con 0x7f834c0f6280 2021-03-09T09:06:39.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f834c0f6280 2021-03-09T09:06:39.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:39.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402a180 con 0x7f834c0f6280 2021-03-09T09:06:39.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:06:39 smithi146 bash[13136]: debug 2021-03-09T09:06:39.411+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1183 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:06:44.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.411+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338013d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:06:44.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.411+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338013d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:44.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.411+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338013d50 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f8338062510 tx=0x7f8344048960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:44.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.411+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338013d50 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f8338062510 tx=0x7f8344048960).stop 2021-03-09T09:06:44.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338013d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:44.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338013d50 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:44.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:44.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380283c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:44.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:44.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022de0 con 0x7f834c0f4690 2021-03-09T09:06:44.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380667b0 con 0x7f834c0f5930 2021-03-09T09:06:44.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338033d70 con 0x7f8338062ca0 2021-03-09T09:06:44.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.411+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:06:44.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.411+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:44.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:06:44.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:44.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380283c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:44.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:44.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:44.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:44.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:44.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338015070 con 0x7f834c0f5930 2021-03-09T09:06:44.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380283c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344048960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:44.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f834c0f5930 2021-03-09T09:06:44.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344020380 con 0x7f834c0f5930 2021-03-09T09:06:44.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:44.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023040 con 0x7f834c0f5930 2021-03-09T09:06:44.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:06:44 smithi146 bash[13136]: debug 2021-03-09T09:06:44.411+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1209 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:06:49.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.411+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f83380283c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:06:49.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.411+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f83380283c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:49.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.411+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380283c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344048960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:49.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.411+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380283c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344048960).stop 2021-03-09T09:06:49.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f83380283c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:49.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f83380283c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:49.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:49.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:49.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:49.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e540 con 0x7f834c0f4690 2021-03-09T09:06:49.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021840 con 0x7f834c0f6280 2021-03-09T09:06:49.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338033d70 con 0x7f8338062ca0 2021-03-09T09:06:49.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.415+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:06:49.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.415+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:49.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:06:49.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:49.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:49.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:49.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:49.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:49.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:49.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005a60 con 0x7f834c0f6280 2021-03-09T09:06:49.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.419+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015540 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:49.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f834c0f6280 2021-03-09T09:06:49.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0222d0 con 0x7f834c0f6280 2021-03-09T09:06:49.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:49.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f834c0f6280 2021-03-09T09:06:49.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:06:49 smithi146 bash[13136]: debug 2021-03-09T09:06:49.415+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1232 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:06:54.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338015540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:06:54.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338015540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:54.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015540 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:54.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015540 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c04e870).stop 2021-03-09T09:06:54.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338015540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:54.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015540 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:54.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:54.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833802df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:54.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:54.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018e60 con 0x7f834c0f4690 2021-03-09T09:06:54.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380667b0 con 0x7f834c0f5930 2021-03-09T09:06:54.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338033d70 con 0x7f8338062ca0 2021-03-09T09:06:54.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:06:54.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:06:54.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:54.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:54.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833802df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:54.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:54.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:54.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:54.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:54.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010a60 con 0x7f834c0f5930 2021-03-09T09:06:54.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.419+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833802df30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f834404e2d0 tx=0x7f834404ac20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:54.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029030 con 0x7f834c0f5930 2021-03-09T09:06:54.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344048830 con 0x7f834c0f5930 2021-03-09T09:06:54.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:54.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023070 con 0x7f834c0f5930 2021-03-09T09:06:54.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:06:54 smithi146 bash[13136]: debug 2021-03-09T09:06:54.415+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1259 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:06:59.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833802df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:06:59.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833802df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:59.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833802df30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f834404e2d0 tx=0x7f834404ac20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:59.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833802df30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f834404e2d0 tx=0x7f834404ac20).stop 2021-03-09T09:06:59.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 msgr2=0x7f833802df30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:59.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f5930 0x7f833802df30 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:59.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:59.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:59.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:59.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380117d0 con 0x7f834c0f4690 2021-03-09T09:06:59.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380074c0 con 0x7f834c0f6280 2021-03-09T09:06:59.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb10 con 0x7f8338062ca0 2021-03-09T09:06:59.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:06:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:06:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:59.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:59.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:59.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:59.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:59.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.415+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064c50 con 0x7f834c0f6280 2021-03-09T09:06:59.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.419+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a7d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f833402e960 tx=0x7f8334028ff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:59.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e040 con 0x7f834c0f6280 2021-03-09T09:06:59.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007c30 con 0x7f834c0f6280 2021-03-09T09:06:59.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:06:59.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402e5d0 con 0x7f834c0f6280 2021-03-09T09:06:59.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:06:59 smithi146 bash[13136]: debug 2021-03-09T09:06:59.415+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1282 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:07:04.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.415+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:07:04.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.415+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:04.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.415+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a7d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f833402e960 tx=0x7f8334028ff0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:04.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.415+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a7d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f833402e960 tx=0x7f8334028ff0).stop 2021-03-09T09:07:04.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801a7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:04.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a7d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.415+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:07:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.415+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380616c0 0x7f8338061ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:04.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022de0 con 0x7f834c0f4690 2021-03-09T09:07:04.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e540 con 0x7f83380616c0 2021-03-09T09:07:04.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338033d70 con 0x7f8338062ca0 2021-03-09T09:07:04.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:07:04.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:04.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.419+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380616c0 0x7f8338061ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:04.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.419+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:04.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.419+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:04.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.419+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:04.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.419+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:04.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.419+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801bcb0 con 0x7f83380616c0 2021-03-09T09:07:04.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.419+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380616c0 0x7f8338061ae0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f834401efd0 tx=0x7f8344070040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:04.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029080 con 0x7f83380616c0 2021-03-09T09:07:04.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344053f00 con 0x7f83380616c0 2021-03-09T09:07:04.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:04.427+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023070 con 0x7f83380616c0 2021-03-09T09:07:04.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:07:04 smithi146 bash[13136]: debug 2021-03-09T09:07:04.415+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1309 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:07:09.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380616c0 msgr2=0x7f8338061ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:07:09.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380616c0 msgr2=0x7f8338061ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:09.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380616c0 0x7f8338061ae0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f834401efd0 tx=0x7f8344070040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:09.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380616c0 0x7f8338061ae0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f834401efd0 tx=0x7f8344070040).stop 2021-03-09T09:07:09.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380616c0 msgr2=0x7f8338061ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:09.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380616c0 0x7f8338061ae0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:09.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:09.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:09.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:09.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061fd0 con 0x7f834c0f4690 2021-03-09T09:07:09.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380074c0 con 0x7f834c0f6280 2021-03-09T09:07:09.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380664c0 con 0x7f8338062ca0 2021-03-09T09:07:09.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.419+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:09.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.419+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:07:09.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.419+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:09.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.419+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:07:09.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.419+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:09.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.419+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:09.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.419+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:09.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.419+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:09.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.419+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:09.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.419+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005a60 con 0x7f834c0f6280 2021-03-09T09:07:09.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.419+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800acf0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f8334028ff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:09.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340462b0 con 0x7f834c0f6280 2021-03-09T09:07:09.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f834c0f6280 2021-03-09T09:07:09.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:09.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402e2d0 con 0x7f834c0f6280 2021-03-09T09:07:09.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:07:09 smithi146 bash[13136]: debug 2021-03-09T09:07:09.415+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1332 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:07:14.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:07:14.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:14.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800acf0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f8334028ff0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:14.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800acf0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f8334028ff0).stop 2021-03-09T09:07:14.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800acf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:14.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800acf0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:14.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:14.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:14.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338006ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:14.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338033d70 con 0x7f834c0f4690 2021-03-09T09:07:14.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629a0 con 0x7f8338006a70 2021-03-09T09:07:14.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012820 con 0x7f8338062ca0 2021-03-09T09:07:14.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:07:14.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:07:14.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:14.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338006ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:14.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:14.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:14.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:14.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:14.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:14.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010a60 con 0x7f8338006a70 2021-03-09T09:07:14.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338006ed0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:14.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c067500 con 0x7f8338006a70 2021-03-09T09:07:14.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04de90 con 0x7f8338006a70 2021-03-09T09:07:14.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:14.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c067130 con 0x7f8338006a70 2021-03-09T09:07:14.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:07:14 smithi146 bash[13136]: debug 2021-03-09T09:07:14.419+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1359 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:07:19.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338006ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:07:19.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338006ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:19.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338006ed0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:19.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338006ed0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04e870).stop 2021-03-09T09:07:19.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338006ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:19.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338006ed0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:19.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:19.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:19.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:19.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338033d70 con 0x7f834c0f4690 2021-03-09T09:07:19.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380259e0 con 0x7f834c0f6280 2021-03-09T09:07:19.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064360 con 0x7f8338062ca0 2021-03-09T09:07:19.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:07:19.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:19.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:07:19.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:19.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.418+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:19.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.422+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:19.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.422+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:19.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.422+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:19.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.422+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:19.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.422+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028fe0 con 0x7f834c0f6280 2021-03-09T09:07:19.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.422+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800acf0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f833c069040 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:19.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0042f0 con 0x7f834c0f6280 2021-03-09T09:07:19.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0716b0 con 0x7f834c0f6280 2021-03-09T09:07:19.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:19.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05fb40 con 0x7f834c0f6280 2021-03-09T09:07:19.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:07:19 smithi146 bash[13136]: debug 2021-03-09T09:07:19.419+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1382 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:07:24.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:07:24.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:24.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800acf0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f833c069040 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:24.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800acf0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f833c069040 tx=0x7f833c04e870).stop 2021-03-09T09:07:24.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800acf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:24.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800acf0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:24.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:24.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833800dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:24.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:24.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064360 con 0x7f834c0f4690 2021-03-09T09:07:24.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:07:24.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cad0 con 0x7f8338006a70 2021-03-09T09:07:24.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064cf0 con 0x7f8338062ca0 2021-03-09T09:07:24.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:24.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:07:24.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:24.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.418+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833800dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:24.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.422+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:24.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.422+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:24.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.422+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:24.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.422+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:24.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.422+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f8338006a70 2021-03-09T09:07:24.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.422+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833800dad0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344070040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:24.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029020 con 0x7f8338006a70 2021-03-09T09:07:24.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344048830 con 0x7f8338006a70 2021-03-09T09:07:24.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:24.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023070 con 0x7f8338006a70 2021-03-09T09:07:24.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:07:24 smithi146 bash[13136]: debug 2021-03-09T09:07:24.418+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1409 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:07:29.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833800dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:07:29.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833800dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:29.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833800dad0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344070040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:29.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833800dad0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344070040).stop 2021-03-09T09:07:29.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833800dad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:29.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833800dad0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:29.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338010c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:29.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:29.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:29.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061010 con 0x7f834c0f4690 2021-03-09T09:07:29.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338033d70 con 0x7f834c0f6280 2021-03-09T09:07:29.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064360 con 0x7f8338062ca0 2021-03-09T09:07:29.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:07:29.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:29.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338010c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:29.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:07:29.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.418+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:29.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.422+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:29.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.422+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:29.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.422+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:29.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.422+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:29.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.422+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005a60 con 0x7f834c0f6280 2021-03-09T09:07:29.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.422+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338010c50 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f8334047000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:29.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e040 con 0x7f834c0f6280 2021-03-09T09:07:29.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401fe20 con 0x7f834c0f6280 2021-03-09T09:07:29.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:29.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334007a10 con 0x7f834c0f6280 2021-03-09T09:07:29.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:07:29 smithi146 bash[13136]: debug 2021-03-09T09:07:29.418+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1432 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:07:34.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.418+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338010c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:07:34.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.418+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338010c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:34.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.418+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338010c50 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f8334047000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:34.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.418+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338010c50 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f8334047000).stop 2021-03-09T09:07:34.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338010c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:34.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338010c50 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:34.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:34.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:34.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.418+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:34.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380276f0 con 0x7f834c0f4690 2021-03-09T09:07:34.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064cf0 con 0x7f8338006a70 2021-03-09T09:07:34.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cad0 con 0x7f8338062ca0 2021-03-09T09:07:34.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.418+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:07:34.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.418+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:34.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.418+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:07:34.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.418+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:34.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.422+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:34.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.422+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:34.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.422+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:34.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.422+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:34.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.422+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:34.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.422+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338033d70 con 0x7f8338006a70 2021-03-09T09:07:34.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.422+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833807dc80 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f833c00a0a0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:34.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068040 con 0x7f8338006a70 2021-03-09T09:07:34.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0528c0 con 0x7f8338006a70 2021-03-09T09:07:34.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:34.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0151a0 con 0x7f8338006a70 2021-03-09T09:07:34.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:07:34 smithi146 bash[13136]: debug 2021-03-09T09:07:34.418+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1459 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:07:39.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.418+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:07:39.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.418+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:39.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.418+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833807dc80 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f833c00a0a0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:39.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.418+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833807dc80 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f833c00a0a0 tx=0x7f833c04e870).stop 2021-03-09T09:07:39.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:39.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833807dc80 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:39.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:39.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:39.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:39.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064360 con 0x7f834c0f4690 2021-03-09T09:07:39.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018750 con 0x7f834c0f6280 2021-03-09T09:07:39.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b280 con 0x7f8338062ca0 2021-03-09T09:07:39.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:07:39.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:39.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:39.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:07:39.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:39.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:39.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:39.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:39.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:39.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a710 con 0x7f834c0f6280 2021-03-09T09:07:39.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.422+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807e170 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f833803eaa0 tx=0x7f8344070040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:39.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f834c0f6280 2021-03-09T09:07:39.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344071300 con 0x7f834c0f6280 2021-03-09T09:07:39.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:39.430+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023070 con 0x7f834c0f6280 2021-03-09T09:07:39.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:07:39 smithi146 bash[13136]: debug 2021-03-09T09:07:39.422+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1482 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:07:44.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:07:44.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:44.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807e170 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f833803eaa0 tx=0x7f8344070040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:44.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807e170 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f833803eaa0 tx=0x7f8344070040).stop 2021-03-09T09:07:44.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807e170 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:44.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807e170 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:44.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:44.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:07:44.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:44.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338011ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:44.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:44.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:07:44.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:44.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380236b0 con 0x7f834c0f4690 2021-03-09T09:07:44.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006400 con 0x7f8338006a70 2021-03-09T09:07:44.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f8338062ca0 2021-03-09T09:07:44.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.422+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338011ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:44.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.426+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:44.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.426+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:44.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.426+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:44.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.426+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:44.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.426+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800b8a0 con 0x7f8338006a70 2021-03-09T09:07:44.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.430+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338011ba0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344070040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:44.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.430+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029020 con 0x7f8338006a70 2021-03-09T09:07:44.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.430+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344013600 con 0x7f8338006a70 2021-03-09T09:07:44.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:44.430+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023070 con 0x7f8338006a70 2021-03-09T09:07:44.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:07:44 smithi146 bash[13136]: debug 2021-03-09T09:07:44.422+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1509 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:07:49.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.422+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338011ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:07:49.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.422+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338011ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:49.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.422+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338011ba0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344070040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:49.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.422+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338011ba0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344070040).stop 2021-03-09T09:07:49.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338011ba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:49.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.422+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338011ba0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:49.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.422+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:49.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.422+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:07:49.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.426+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:49.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.426+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:49.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.426+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:49.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.426+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:49.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012b40 con 0x7f834c0f4690 2021-03-09T09:07:49.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005d20 con 0x7f834c0f6280 2021-03-09T09:07:49.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016770 con 0x7f8338062ca0 2021-03-09T09:07:49.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.426+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:07:49.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.426+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:49.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.430+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:49.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.430+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:49.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.430+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:49.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.430+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:49.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.430+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022220 con 0x7f834c0f6280 2021-03-09T09:07:49.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.430+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801e230 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f834400e480 tx=0x7f8344070040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:49.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.434+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029020 con 0x7f834c0f6280 2021-03-09T09:07:49.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.434+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344054980 con 0x7f834c0f6280 2021-03-09T09:07:49.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:49.434+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440770f0 con 0x7f834c0f6280 2021-03-09T09:07:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:07:49 smithi146 bash[13136]: debug 2021-03-09T09:07:49.422+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1532 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:07:54.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:07:54.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:54.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801e230 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f834400e480 tx=0x7f8344070040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:54.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801e230 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f834400e480 tx=0x7f8344070040).stop 2021-03-09T09:07:54.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801e230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:54.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801e230 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:54.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:54.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:07:54.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:54.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:54.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:54.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380712d0 con 0x7f834c0f4690 2021-03-09T09:07:54.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380236b0 con 0x7f8338006a70 2021-03-09T09:07:54.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ec90 con 0x7f8338062ca0 2021-03-09T09:07:54.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:07:54.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:54.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:54.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:54.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:54.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:54.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:54.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f8338006a70 2021-03-09T09:07:54.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833807dc80 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:54.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f8338006a70 2021-03-09T09:07:54.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.430+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c034410 con 0x7f8338006a70 2021-03-09T09:07:54.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:54.430+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338006a70 2021-03-09T09:07:54.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:07:54 smithi146 bash[13136]: debug 2021-03-09T09:07:54.426+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1558 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:07:59.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.422+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:07:59.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.422+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:59.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.422+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833807dc80 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:59.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.422+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833807dc80 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c04e870).stop 2021-03-09T09:07:59.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.422+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:59.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.422+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833807dc80 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:59.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.426+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:59.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.426+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:59.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.426+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:59.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.426+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:59.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f834c0f4690 2021-03-09T09:07:59.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024d80 con 0x7f834c0f6280 2021-03-09T09:07:59.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a8c0 con 0x7f8338062ca0 2021-03-09T09:07:59.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.426+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:07:59.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.426+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:59.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.426+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:07:59.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.426+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.430+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.430+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.430+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.430+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:59.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.430+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005a60 con 0x7f834c0f6280 2021-03-09T09:07:59.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.430+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800c560 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f8334028460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:59.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.434+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e040 con 0x7f834c0f6280 2021-03-09T09:07:59.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.434+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334032050 con 0x7f834c0f6280 2021-03-09T09:07:59.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:07:59.434+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340090d0 con 0x7f834c0f6280 2021-03-09T09:07:59.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:07:59 smithi146 bash[13136]: debug 2021-03-09T09:07:59.426+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1582 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:08:04.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:08:04.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:04.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800c560 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f8334028460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:04.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800c560 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f8334028460).stop 2021-03-09T09:08:04.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800c560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:04.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800c560 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:04.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:04.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:04.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338015d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:04.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e100 con 0x7f834c0f4690 2021-03-09T09:08:04.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021c80 con 0x7f8338006a70 2021-03-09T09:08:04.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b280 con 0x7f8338062ca0 2021-03-09T09:08:04.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:08:04.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:04.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:08:04.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:04.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338015d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:04.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:04.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:04.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:04.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:04.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.426+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806c8d0 con 0x7f8338006a70 2021-03-09T09:08:04.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.430+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338015d80 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f833c052580 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:04.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.430+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f8338006a70 2021-03-09T09:08:04.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.430+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0521d0 con 0x7f8338006a70 2021-03-09T09:08:04.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:04.434+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c060d80 con 0x7f8338006a70 2021-03-09T09:08:04.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:08:04 smithi146 bash[13136]: debug 2021-03-09T09:08:04.426+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1608 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:08:09.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.425+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338015d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:08:09.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.425+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338015d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:09.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.425+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338015d80 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f833c052580 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:09.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.425+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338015d80 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f833c052580 tx=0x7f833c04e870).stop 2021-03-09T09:08:09.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338015d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:09.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338015d80 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:09.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:09.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:09.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:09.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064cf0 con 0x7f834c0f4690 2021-03-09T09:08:09.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f834c0f6280 2021-03-09T09:08:09.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e100 con 0x7f8338062ca0 2021-03-09T09:08:09.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.425+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:08:09.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.425+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:09.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.425+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:08:09.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.425+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:09.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.429+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:09.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.429+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:09.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.429+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:09.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.429+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:09.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.429+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:09.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.429+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380156f0 con 0x7f834c0f6280 2021-03-09T09:08:09.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.429+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344070040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:09.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f834c0f6280 2021-03-09T09:08:09.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834401e450 con 0x7f834c0f6280 2021-03-09T09:08:09.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:09.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023070 con 0x7f834c0f6280 2021-03-09T09:08:09.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:08:09 smithi146 bash[13136]: debug 2021-03-09T09:08:09.426+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1632 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:08:14.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:08:14.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:14.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344070040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:14.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344070040).stop 2021-03-09T09:08:14.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:14.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:14.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:14.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338009fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:14.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:14.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175c0 con 0x7f834c0f4690 2021-03-09T09:08:14.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027da0 con 0x7f8338006a70 2021-03-09T09:08:14.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019060 con 0x7f8338062ca0 2021-03-09T09:08:14.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:08:14.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:14.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:08:14.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:14.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.425+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338009fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:14.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.429+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:14.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.429+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:14.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.429+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:14.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.429+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:14.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.429+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338013690 con 0x7f8338006a70 2021-03-09T09:08:14.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.429+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338009fe0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f834401f870 tx=0x7f8344070040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:14.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344072db0 con 0x7f8338006a70 2021-03-09T09:08:14.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440133f0 con 0x7f8338006a70 2021-03-09T09:08:14.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:14.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344020ce0 con 0x7f8338006a70 2021-03-09T09:08:14.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:08:14 smithi146 bash[13136]: debug 2021-03-09T09:08:14.426+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1658 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:08:19.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.425+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338009fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:08:19.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.425+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338009fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:19.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.425+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338009fe0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f834401f870 tx=0x7f8344070040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:19.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.425+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338009fe0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f834401f870 tx=0x7f8344070040).stop 2021-03-09T09:08:19.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338009fe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:19.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338009fe0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:19.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:19.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.425+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:08:19.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.425+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:19.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:19.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:19.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e100 con 0x7f834c0f4690 2021-03-09T09:08:19.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a160 con 0x7f834c0f6280 2021-03-09T09:08:19.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380051c0 con 0x7f8338062ca0 2021-03-09T09:08:19.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.429+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:08:19.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.429+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:19.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.429+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.429+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.429+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.429+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.429+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.429+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380114e0 con 0x7f834c0f6280 2021-03-09T09:08:19.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.429+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023ce0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:19.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f834c0f6280 2021-03-09T09:08:19.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00d350 con 0x7f834c0f6280 2021-03-09T09:08:19.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:19.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d280 con 0x7f834c0f6280 2021-03-09T09:08:19.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:08:19 smithi146 bash[13136]: debug 2021-03-09T09:08:19.430+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1682 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:08:24.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.425+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338023ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:08:24.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.425+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338023ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:24.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.425+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023ce0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:24.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.425+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023ce0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c04e870).stop 2021-03-09T09:08:24.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338023ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:24.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023ce0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:24.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:24.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:24.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:24.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014f20 con 0x7f834c0f4690 2021-03-09T09:08:24.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012b40 con 0x7f8338006a70 2021-03-09T09:08:24.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f8338062ca0 2021-03-09T09:08:24.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.429+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:08:24.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.429+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:08:24.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.429+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:24.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.429+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:24.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.429+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:24.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.433+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:24.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.433+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:24.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.433+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:24.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.433+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:24.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.433+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f8338006a70 2021-03-09T09:08:24.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.437+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802c100 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f833c02bf30 tx=0x7f833c06f040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:24.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04d4d0 con 0x7f8338006a70 2021-03-09T09:08:24.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c071a80 con 0x7f8338006a70 2021-03-09T09:08:24.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:24.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f8338006a70 2021-03-09T09:08:24.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:08:24 smithi146 bash[13136]: debug 2021-03-09T09:08:24.430+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1708 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:08:29.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:08:29.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:29.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802c100 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f833c02bf30 tx=0x7f833c06f040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:29.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802c100 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f833c02bf30 tx=0x7f833c06f040).stop 2021-03-09T09:08:29.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802c100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:29.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802c100 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:29.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:29.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:29.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:29.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f560 con 0x7f834c0f4690 2021-03-09T09:08:29.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380686a0 con 0x7f834c0f6280 2021-03-09T09:08:29.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806db20 con 0x7f8338062ca0 2021-03-09T09:08:29.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:08:29.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:29.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:08:29.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:29.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.429+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:29.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:29.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:29.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:29.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:29.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380236b0 con 0x7f834c0f6280 2021-03-09T09:08:29.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023ba0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334028460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:29.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e040 con 0x7f834c0f6280 2021-03-09T09:08:29.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334008480 con 0x7f834c0f6280 2021-03-09T09:08:29.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:29.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334008000 con 0x7f834c0f6280 2021-03-09T09:08:29.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:08:29 smithi146 bash[13136]: debug 2021-03-09T09:08:29.430+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1732 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:08:34.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338023ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:08:34.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338023ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:34.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023ba0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334028460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:34.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023ba0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334028460).stop 2021-03-09T09:08:34.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338023ba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:34.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338023ba0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:34.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:34.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:34.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:34.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026a80 con 0x7f834c0f4690 2021-03-09T09:08:34.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f560 con 0x7f8338006a70 2021-03-09T09:08:34.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806db20 con 0x7f8338062ca0 2021-03-09T09:08:34.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:08:34.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:34.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:34.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:08:34.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.429+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:34.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:34.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:34.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:34.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:34.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010a60 con 0x7f8338006a70 2021-03-09T09:08:34.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f8334000e50 tx=0x7f8334028460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:34.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e040 con 0x7f8338006a70 2021-03-09T09:08:34.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334050350 con 0x7f8338006a70 2021-03-09T09:08:34.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:34.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334029c00 con 0x7f8338006a70 2021-03-09T09:08:34.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:08:34 smithi146 bash[13136]: debug 2021-03-09T09:08:34.430+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1757 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:08:39.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.429+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:08:39.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.429+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:39.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.429+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f8334000e50 tx=0x7f8334028460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:39.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.429+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f8334000e50 tx=0x7f8334028460).stop 2021-03-09T09:08:39.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:39.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:39.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:39.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:39.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:39.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014f20 con 0x7f834c0f4690 2021-03-09T09:08:39.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380238b0 con 0x7f834c0f6280 2021-03-09T09:08:39.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f560 con 0x7f8338062ca0 2021-03-09T09:08:39.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.429+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:08:39.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.429+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:39.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.429+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:08:39.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.433+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:39.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338024560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:39.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:39.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:39.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:39.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:39.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024fc0 con 0x7f834c0f6280 2021-03-09T09:08:39.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338024560 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f833400a1c0 tx=0x7f8334028460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:39.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f834c0f6280 2021-03-09T09:08:39.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334001e80 con 0x7f834c0f6280 2021-03-09T09:08:39.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:39.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340017a0 con 0x7f834c0f6280 2021-03-09T09:08:39.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:08:39 smithi146 bash[13136]: debug 2021-03-09T09:08:39.430+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1782 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:08:44.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338024560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:08:44.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338024560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:44.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338024560 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f833400a1c0 tx=0x7f8334028460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:44.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338024560 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f833400a1c0 tx=0x7f8334028460).stop 2021-03-09T09:08:44.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338024560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:44.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338024560 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:44.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:44.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:44.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:44.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:08:44.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:44.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:08:44.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:44.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802db20 con 0x7f834c0f4690 2021-03-09T09:08:44.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380178d0 con 0x7f8338006a70 2021-03-09T09:08:44.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ee30 con 0x7f8338062ca0 2021-03-09T09:08:44.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:44.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:44.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:44.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:44.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:44.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f8338006a70 2021-03-09T09:08:44.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.433+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:44.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068040 con 0x7f8338006a70 2021-03-09T09:08:44.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c051170 con 0x7f8338006a70 2021-03-09T09:08:44.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:44.441+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04de90 con 0x7f8338006a70 2021-03-09T09:08:44.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:08:44 smithi146 bash[13136]: debug 2021-03-09T09:08:44.434+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1807 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:08:49.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:08:49.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:49.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:49.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c0512e0).stop 2021-03-09T09:08:49.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:49.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:49.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:49.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:49.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:49.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806db20 con 0x7f834c0f4690 2021-03-09T09:08:49.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380686a0 con 0x7f834c0f6280 2021-03-09T09:08:49.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f560 con 0x7f8338062ca0 2021-03-09T09:08:49.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:08:49.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:49.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:49.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:49.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:49.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:49.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:08:49.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:49.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:49.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338027540 con 0x7f834c0f6280 2021-03-09T09:08:49.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.433+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f833c013d10 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:49.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f834c0f6280 2021-03-09T09:08:49.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f834c0f6280 2021-03-09T09:08:49.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:49.441+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c071a80 con 0x7f834c0f6280 2021-03-09T09:08:49.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:08:49 smithi146 bash[13136]: debug 2021-03-09T09:08:49.433+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1832 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:08:54.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:08:54.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:54.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f833c013d10 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:54.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f833c013d10 tx=0x7f833c0512e0).stop 2021-03-09T09:08:54.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:54.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:54.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:54.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:54.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:08:54.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:54.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:54.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061250 con 0x7f834c0f4690 2021-03-09T09:08:54.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008ba0 con 0x7f8338006a70 2021-03-09T09:08:54.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cd20 con 0x7f8338062ca0 2021-03-09T09:08:54.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:08:54.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:54.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:54.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:54.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:54.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:54.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:54.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338011f20 con 0x7f8338006a70 2021-03-09T09:08:54.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7f8334008950 tx=0x7f83340283f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:54.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f8338006a70 2021-03-09T09:08:54.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338006a70 2021-03-09T09:08:54.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:54.441+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334007900 con 0x7f8338006a70 2021-03-09T09:08:54.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:08:54 smithi146 bash[13136]: debug 2021-03-09T09:08:54.433+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1857 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:08:59.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:08:59.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:59.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7f8334008950 tx=0x7f83340283f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:59.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7f8334008950 tx=0x7f83340283f0).stop 2021-03-09T09:08:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338004fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:59.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:59.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021c80 con 0x7f834c0f4690 2021-03-09T09:08:59.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071310 con 0x7f834c0f6280 2021-03-09T09:08:59.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007390 con 0x7f8338062ca0 2021-03-09T09:08:59.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.433+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:08:59.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.433+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:59.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:08:59.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:59.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.437+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338004fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:59.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.437+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:59.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.437+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:59.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.437+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:59.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.437+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:59.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.437+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338027d50 con 0x7f834c0f6280 2021-03-09T09:08:59.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.437+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338004fe0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7f83440272b0 tx=0x7f8344070040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:59.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f834c0f6280 2021-03-09T09:08:59.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440270d0 con 0x7f834c0f6280 2021-03-09T09:08:59.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:08:59.441+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344016120 con 0x7f834c0f6280 2021-03-09T09:08:59.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:08:59 smithi146 bash[13136]: debug 2021-03-09T09:08:59.433+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1882 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:09:04.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.433+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338004fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:09:04.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.433+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338004fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:04.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.433+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338004fe0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7f83440272b0 tx=0x7f8344070040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:04.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.433+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338004fe0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7f83440272b0 tx=0x7f8344070040).stop 2021-03-09T09:09:04.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338004fe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:04.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338004fe0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:04.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338025620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:04.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:04.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:04.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a360 con 0x7f834c0f4690 2021-03-09T09:09:04.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026c80 con 0x7f8338006a70 2021-03-09T09:09:04.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016770 con 0x7f8338062ca0 2021-03-09T09:09:04.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.433+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:09:04.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.437+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:04.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.437+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:09:04.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.437+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:04.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.437+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338025620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:04.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.437+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:04.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.437+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:04.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.437+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:04.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.437+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:04.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.437+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065440 con 0x7f8338006a70 2021-03-09T09:09:04.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.437+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338025620 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7f8334030c90 tx=0x7f8334015040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:04.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.441+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334017340 con 0x7f8338006a70 2021-03-09T09:09:04.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.441+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340306c0 con 0x7f8338006a70 2021-03-09T09:09:04.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:04.445+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340077b0 con 0x7f8338006a70 2021-03-09T09:09:04.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:09:04 smithi146 bash[13136]: debug 2021-03-09T09:09:04.437+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1907 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:09:09.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338025620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:09:09.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338025620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:09.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338025620 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7f8334030c90 tx=0x7f8334015040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:09.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338025620 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7f8334030c90 tx=0x7f8334015040).stop 2021-03-09T09:09:09.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338025620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:09.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338025620 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:09.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:09.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:09.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:09.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:09:09.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:09.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380686a0 con 0x7f834c0f4690 2021-03-09T09:09:09.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068800 con 0x7f834c0f6280 2021-03-09T09:09:09.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019060 con 0x7f8338062ca0 2021-03-09T09:09:09.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:09:09.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:09.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.436+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:09.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.440+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:09.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.440+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:09.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.440+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:09.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.440+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:09.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.440+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007510 con 0x7f834c0f6280 2021-03-09T09:09:09.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.440+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800f130 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f8338006a00 tx=0x7f8344070040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:09.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f834c0f6280 2021-03-09T09:09:09.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405de90 con 0x7f834c0f6280 2021-03-09T09:09:09.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:09.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023030 con 0x7f834c0f6280 2021-03-09T09:09:09.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:09:09 smithi146 bash[13136]: debug 2021-03-09T09:09:09.437+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1933 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:09:14.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:09:14.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800f130 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f8338006a00 tx=0x7f8344070040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800f130 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f8338006a00 tx=0x7f8344070040).stop 2021-03-09T09:09:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800f130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:14.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800f130 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:14.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:14.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:14.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:14.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380340f0 con 0x7f834c0f4690 2021-03-09T09:09:14.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800baf0 con 0x7f8338006a70 2021-03-09T09:09:14.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380153c0 con 0x7f8338062ca0 2021-03-09T09:09:14.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:09:14.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:14.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:09:14.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:14.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.436+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:14.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.440+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:14.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:14.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.440+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:14.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:14.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.440+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f8338006a70 2021-03-09T09:09:14.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f833400a3f0 tx=0x7f8334015040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:14.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e080 con 0x7f8338006a70 2021-03-09T09:09:14.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833406a000 con 0x7f8338006a70 2021-03-09T09:09:14.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:14.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334008650 con 0x7f8338006a70 2021-03-09T09:09:14.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:09:14 smithi146 bash[13136]: debug 2021-03-09T09:09:14.437+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1957 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:09:19.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:09:19.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:19.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f833400a3f0 tx=0x7f8334015040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:19.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f833400a3f0 tx=0x7f8334015040).stop 2021-03-09T09:09:19.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:19.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:19.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:19.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:19.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:19.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:09:19.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:19.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:09:19.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:19.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027710 con 0x7f834c0f4690 2021-03-09T09:09:19.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012b40 con 0x7f834c0f6280 2021-03-09T09:09:19.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023de0 con 0x7f8338062ca0 2021-03-09T09:09:19.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:19.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:19.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:19.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:19.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:19.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380114e0 con 0x7f834c0f6280 2021-03-09T09:09:19.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7f833402f7b0 tx=0x7f8334015040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:19.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e030 con 0x7f834c0f6280 2021-03-09T09:09:19.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334017130 con 0x7f834c0f6280 2021-03-09T09:09:19.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:19.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340079c0 con 0x7f834c0f6280 2021-03-09T09:09:19.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:09:19 smithi146 bash[13136]: debug 2021-03-09T09:09:19.437+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 1983 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:09:24.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:09:24.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:24.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7f833402f7b0 tx=0x7f8334015040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:24.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7f833402f7b0 tx=0x7f8334015040).stop 2021-03-09T09:09:24.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:24.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:24.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:24.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:24.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:24.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:09:24.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:24.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:09:24.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:24.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380340f0 con 0x7f834c0f4690 2021-03-09T09:09:24.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380121b0 con 0x7f8338006a70 2021-03-09T09:09:24.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024fb0 con 0x7f8338062ca0 2021-03-09T09:09:24.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:24.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:24.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:24.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:24.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:24.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f8338006a70 2021-03-09T09:09:24.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7f8334004f00 tx=0x7f833405e040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:24.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f8338006a70 2021-03-09T09:09:24.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833400a000 con 0x7f8338006a70 2021-03-09T09:09:24.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:24.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833406cb20 con 0x7f8338006a70 2021-03-09T09:09:24.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:09:24 smithi146 bash[13136]: debug 2021-03-09T09:09:24.441+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2007 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:09:29.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:09:29.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:29.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7f8334004f00 tx=0x7f833405e040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:29.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7f8334004f00 tx=0x7f833405e040).stop 2021-03-09T09:09:29.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:29.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:29.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:29.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:29.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:29.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027710 con 0x7f834c0f4690 2021-03-09T09:09:29.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380340f0 con 0x7f834c0f6280 2021-03-09T09:09:29.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011730 con 0x7f8338062ca0 2021-03-09T09:09:29.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:09:29.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:29.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:09:29.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:29.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.440+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:29.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.444+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:29.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.444+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:29.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.444+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:29.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.444+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:29.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.444+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380675e0 con 0x7f834c0f6280 2021-03-09T09:09:29.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.444+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833405e040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:29.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e030 con 0x7f834c0f6280 2021-03-09T09:09:29.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404a110 con 0x7f834c0f6280 2021-03-09T09:09:29.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:29.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334060d70 con 0x7f834c0f6280 2021-03-09T09:09:29.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:09:29 smithi146 bash[13136]: debug 2021-03-09T09:09:29.441+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2033 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:09:34.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:09:34.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:34.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833405e040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:34.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833405e040).stop 2021-03-09T09:09:34.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:34.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:34.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:34.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:34.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:34.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801eb30 con 0x7f834c0f4690 2021-03-09T09:09:34.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380662e0 con 0x7f8338006a70 2021-03-09T09:09:34.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015740 con 0x7f8338062ca0 2021-03-09T09:09:34.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:09:34.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:34.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:34.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:09:34.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:34.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:34.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:34.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:34.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:34.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007510 con 0x7f8338006a70 2021-03-09T09:09:34.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f833c0194c0 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:34.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338006a70 2021-03-09T09:09:34.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01acd0 con 0x7f8338006a70 2021-03-09T09:09:34.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:34.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338006a70 2021-03-09T09:09:34.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:09:34 smithi146 bash[13136]: debug 2021-03-09T09:09:34.441+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2058 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:09:39.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.440+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:09:39.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.440+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:39.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.440+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f833c0194c0 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:39.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.440+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f833c0194c0 tx=0x7f833c0512e0).stop 2021-03-09T09:09:39.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:39.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.440+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:39.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.440+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:39.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.440+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:39.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.440+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:39.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380121b0 con 0x7f834c0f4690 2021-03-09T09:09:39.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380340f0 con 0x7f834c0f6280 2021-03-09T09:09:39.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.440+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:09:39.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.440+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:39.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.444+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:09:39.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.444+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:39.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.444+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:39.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cd20 con 0x7f8338062ca0 2021-03-09T09:09:39.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.444+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:39.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.444+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:39.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.444+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:39.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.444+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:39.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.444+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380137f0 con 0x7f834c0f6280 2021-03-09T09:09:39.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.444+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f834402c040 tx=0x7f834405e000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:39.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f834c0f6280 2021-03-09T09:09:39.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f834c0f6280 2021-03-09T09:09:39.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:39.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023070 con 0x7f834c0f6280 2021-03-09T09:09:39.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:09:39 smithi146 bash[13136]: debug 2021-03-09T09:09:39.441+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2082 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:09:44.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:09:44.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:44.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f834402c040 tx=0x7f834405e000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:44.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f834402c040 tx=0x7f834405e000).stop 2021-03-09T09:09:44.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:44.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:44.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:44.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:44.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:44.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:09:44.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:44.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:09:44.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:44.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024fb0 con 0x7f834c0f4690 2021-03-09T09:09:44.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011f00 con 0x7f8338006a70 2021-03-09T09:09:44.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f400 con 0x7f8338062ca0 2021-03-09T09:09:44.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:44.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:44.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:44.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:44.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:44.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.444+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f8338006a70 2021-03-09T09:09:44.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.448+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7f833c070930 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:44.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c070590 con 0x7f8338006a70 2021-03-09T09:09:44.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0703b0 con 0x7f8338006a70 2021-03-09T09:09:44.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:44.452+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338006a70 2021-03-09T09:09:44.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:09:44 smithi146 bash[13136]: debug 2021-03-09T09:09:44.441+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2108 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:09:49.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:09:49.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:49.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7f833c070930 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:49.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7f833c070930 tx=0x7f833c0512e0).stop 2021-03-09T09:09:49.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:49.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:49.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:49.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:49.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:49.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801eb30 con 0x7f834c0f4690 2021-03-09T09:09:49.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020ab0 con 0x7f834c0f6280 2021-03-09T09:09:49.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380340f0 con 0x7f8338062ca0 2021-03-09T09:09:49.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:09:49.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:49.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:49.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:09:49.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.444+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:49.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.448+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:49.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.448+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:49.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.448+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:49.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.448+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:49.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.448+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380689b0 con 0x7f834c0f6280 2021-03-09T09:09:49.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.448+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800f130 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7f8338009c80 tx=0x7f834405e000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:49.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f834c0f6280 2021-03-09T09:09:49.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834401f330 con 0x7f834c0f6280 2021-03-09T09:09:49.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:49.452+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023070 con 0x7f834c0f6280 2021-03-09T09:09:49.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:09:49 smithi146 bash[13136]: debug 2021-03-09T09:09:49.445+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2131 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:09:54.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:09:54.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:54.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800f130 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7f8338009c80 tx=0x7f834405e000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:54.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800f130 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7f8338009c80 tx=0x7f834405e000).stop 2021-03-09T09:09:54.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800f130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:54.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800f130 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:54.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:54.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:54.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:09:54.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:54.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:54.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806db20 con 0x7f834c0f4690 2021-03-09T09:09:54.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024fb0 con 0x7f8338006a70 2021-03-09T09:09:54.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380340f0 con 0x7f8338062ca0 2021-03-09T09:09:54.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:09:54.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:54.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:54.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:54.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:54.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:54.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:54.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338016740 con 0x7f8338006a70 2021-03-09T09:09:54.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404c530).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:54.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344013540 con 0x7f8338006a70 2021-03-09T09:09:54.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405ee30 con 0x7f8338006a70 2021-03-09T09:09:54.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:54.452+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344017c50 con 0x7f8338006a70 2021-03-09T09:09:54.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:09:54 smithi146 bash[13136]: debug 2021-03-09T09:09:54.449+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2157 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:09:59.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:09:59.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:59.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404c530).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:59.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404c530).stop 2021-03-09T09:09:59.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:59.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:59.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380648c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:59.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:59.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:59.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f834c0f4690 2021-03-09T09:09:59.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380662e0 con 0x7f834c0f6280 2021-03-09T09:09:59.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380340f0 con 0x7f8338062ca0 2021-03-09T09:09:59.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:09:59.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:59.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:09:59.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380648c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:59.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:59.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:59.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:59.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:59.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:59.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338004950 con 0x7f834c0f6280 2021-03-09T09:09:59.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380648c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7f8334016010 tx=0x7f833405e040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:59.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f834c0f6280 2021-03-09T09:09:59.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.448+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402cd10 con 0x7f834c0f6280 2021-03-09T09:09:59.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:09:59.452+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334060d70 con 0x7f834c0f6280 2021-03-09T09:09:59.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:09:59 smithi146 bash[13136]: debug 2021-03-09T09:09:59.449+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2184 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:10:04.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380648c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:10:04.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380648c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:04.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380648c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7f8334016010 tx=0x7f833405e040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:04.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380648c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7f8334016010 tx=0x7f833405e040).stop 2021-03-09T09:10:04.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380648c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:04.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380648c0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:04.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:04.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:04.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:04.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f834c0f4690 2021-03-09T09:10:04.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024fb0 con 0x7f8338006a70 2021-03-09T09:10:04.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f8338062ca0 2021-03-09T09:10:04.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:10:04.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:04.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:04.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:10:04.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:04.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:04.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:04.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:04.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:04.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338016740 con 0x7f8338006a70 2021-03-09T09:10:04.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f833c078aa0 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:04.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068030 con 0x7f8338006a70 2021-03-09T09:10:04.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.447+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0701d0 con 0x7f8338006a70 2021-03-09T09:10:04.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:04.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338006a70 2021-03-09T09:10:04.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:10:04 smithi146 bash[13136]: debug 2021-03-09T09:10:04.449+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2207 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:10:09.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.447+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:10:09.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.447+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:09.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.447+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f833c078aa0 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:09.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.447+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f833c078aa0 tx=0x7f833c0512e0).stop 2021-03-09T09:10:09.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:09.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:09.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:09.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:09.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:09.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380662e0 con 0x7f834c0f4690 2021-03-09T09:10:09.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019380 con 0x7f834c0f6280 2021-03-09T09:10:09.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024fb0 con 0x7f8338062ca0 2021-03-09T09:10:09.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:10:09.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:09.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:09.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:10:09.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:09.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:09.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:09.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:09.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:09.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380259e0 con 0x7f834c0f6280 2021-03-09T09:10:09.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.451+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7f83440708e0 tx=0x7f834404c530).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:09.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404c5f0 con 0x7f834c0f6280 2021-03-09T09:10:09.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834407ad90 con 0x7f834c0f6280 2021-03-09T09:10:09.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:09.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344049710 con 0x7f834c0f6280 2021-03-09T09:10:09.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:10:09 smithi146 bash[13136]: debug 2021-03-09T09:10:09.449+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2231 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:10:14.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338007d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:10:14.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338007d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:14.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7f83440708e0 tx=0x7f834404c530).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:14.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7f83440708e0 tx=0x7f834404c530).stop 2021-03-09T09:10:14.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338007d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:14.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:14.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:14.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:14.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:14.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024fb0 con 0x7f834c0f4690 2021-03-09T09:10:14.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802db40 con 0x7f8338006a70 2021-03-09T09:10:14.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380157e0 con 0x7f8338062ca0 2021-03-09T09:10:14.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:10:14.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:14.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:10:14.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:14.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:14.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.451+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:14.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:14.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:14.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:14.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380340f0 con 0x7f8338006a70 2021-03-09T09:10:14.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:14.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068030 con 0x7f8338006a70 2021-03-09T09:10:14.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0701d0 con 0x7f8338006a70 2021-03-09T09:10:14.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:14.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338006a70 2021-03-09T09:10:14.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:10:14 smithi146 bash[13136]: debug 2021-03-09T09:10:14.448+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2258 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:10:19.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.451+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:10:19.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.451+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:19.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.451+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:19.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.451+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c0512e0).stop 2021-03-09T09:10:19.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:19.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:19.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:19.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:19.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:19.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023de0 con 0x7f834c0f4690 2021-03-09T09:10:19.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f834c0f6280 2021-03-09T09:10:19.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024fb0 con 0x7f8338062ca0 2021-03-09T09:10:19.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.455+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:10:19.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.455+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:19.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.455+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:19.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:10:19.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:19.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.455+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:19.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.455+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:19.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.455+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:19.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.455+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:19.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.455+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801eae0 con 0x7f834c0f6280 2021-03-09T09:10:19.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.455+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f833402aac0 tx=0x7f833405e040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:19.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833402fb20 con 0x7f834c0f6280 2021-03-09T09:10:19.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402f480 con 0x7f834c0f6280 2021-03-09T09:10:19.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:19.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401fa90 con 0x7f834c0f6280 2021-03-09T09:10:19.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:10:19 smithi146 bash[13136]: debug 2021-03-09T09:10:19.448+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2281 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:10:24.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.451+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338007d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:10:24.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.451+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338007d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:24.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.451+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f833402aac0 tx=0x7f833405e040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:24.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.451+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f833402aac0 tx=0x7f833405e040).stop 2021-03-09T09:10:24.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338007d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:24.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:24.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:24.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:24.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:24.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012b40 con 0x7f834c0f4690 2021-03-09T09:10:24.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011f00 con 0x7f8338006a70 2021-03-09T09:10:24.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8338062ca0 2021-03-09T09:10:24.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.451+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:10:24.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.451+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:24.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.451+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:10:24.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.451+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:24.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.455+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:24.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.455+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:24.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.455+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:24.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.455+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:24.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.455+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:24.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.455+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801eae0 con 0x7f8338006a70 2021-03-09T09:10:24.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.455+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f834405a000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:24.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029040 con 0x7f8338006a70 2021-03-09T09:10:24.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834407ad60 con 0x7f8338006a70 2021-03-09T09:10:24.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:24.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405db80 con 0x7f8338006a70 2021-03-09T09:10:24.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:10:24 smithi146 bash[13136]: debug 2021-03-09T09:10:24.452+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2307 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:10:29.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.451+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:10:29.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.451+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:29.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.451+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f834405a000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:29.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.451+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f834405a000).stop 2021-03-09T09:10:29.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:29.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:29.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:29.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:29.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:29.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380157e0 con 0x7f834c0f4690 2021-03-09T09:10:29.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027310 con 0x7f834c0f6280 2021-03-09T09:10:29.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a10 con 0x7f8338062ca0 2021-03-09T09:10:29.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:10:29.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:10:29.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:29.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:29.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:29.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:29.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:29.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:29.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:29.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338003470 con 0x7f834c0f6280 2021-03-09T09:10:29.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7f833c009f90 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:29.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0067d0 con 0x7f834c0f6280 2021-03-09T09:10:29.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f834c0f6280 2021-03-09T09:10:29.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:29.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d210 con 0x7f834c0f6280 2021-03-09T09:10:29.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:10:29 smithi146 bash[13136]: debug 2021-03-09T09:10:29.452+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2330 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:10:34.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:10:34.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:34.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7f833c009f90 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:34.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7f833c009f90 tx=0x7f833c0512e0).stop 2021-03-09T09:10:34.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:34.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:34.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:34.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:34.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:34.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cf20 con 0x7f834c0f4690 2021-03-09T09:10:34.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e090 con 0x7f8338006a70 2021-03-09T09:10:34.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011f00 con 0x7f8338062ca0 2021-03-09T09:10:34.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:10:34.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:34.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:10:34.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:34.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:34.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:34.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:34.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:34.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:34.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801eae0 con 0x7f8338006a70 2021-03-09T09:10:34.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0x7f833c05ebd0 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:34.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068030 con 0x7f8338006a70 2021-03-09T09:10:34.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c055610 con 0x7f8338006a70 2021-03-09T09:10:34.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:34.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f8338006a70 2021-03-09T09:10:34.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:10:34 smithi146 bash[13136]: debug 2021-03-09T09:10:34.456+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2357 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:10:39.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:10:39.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:39.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0x7f833c05ebd0 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:39.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0x7f833c05ebd0 tx=0x7f833c0512e0).stop 2021-03-09T09:10:39.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:39.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:39.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:39.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:39.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:39.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a10 con 0x7f834c0f4690 2021-03-09T09:10:39.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f834c0f6280 2021-03-09T09:10:39.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028210 con 0x7f8338062ca0 2021-03-09T09:10:39.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:10:39.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:10:39.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:39.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:39.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:39.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:39.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:39.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:39.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:39.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338021e50 con 0x7f834c0f6280 2021-03-09T09:10:39.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d260 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7f833c078700 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:39.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0067d0 con 0x7f834c0f6280 2021-03-09T09:10:39.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f834c0f6280 2021-03-09T09:10:39.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:39.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c034200 con 0x7f834c0f6280 2021-03-09T09:10:39.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:10:39 smithi146 bash[13136]: debug 2021-03-09T09:10:39.452+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2380 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:10:44.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:10:44.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:44.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d260 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7f833c078700 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:44.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d260 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7f833c078700 tx=0x7f833c0512e0).stop 2021-03-09T09:10:44.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802d260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:44.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d260 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:44.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:44.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:44.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:44.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380664e0 con 0x7f834c0f4690 2021-03-09T09:10:44.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380254a0 con 0x7f8338006a70 2021-03-09T09:10:44.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e090 con 0x7f8338062ca0 2021-03-09T09:10:44.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.455+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:10:44.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.455+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:44.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.455+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:10:44.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.455+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:44.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:44.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:44.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:44.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:44.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:44.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801eae0 con 0x7f8338006a70 2021-03-09T09:10:44.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7f833c006a90 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:44.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c034c70 con 0x7f8338006a70 2021-03-09T09:10:44.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c060e90 con 0x7f8338006a70 2021-03-09T09:10:44.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:44.467+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0055f0 con 0x7f8338006a70 2021-03-09T09:10:44.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:10:44 smithi146 bash[13136]: debug 2021-03-09T09:10:44.452+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2407 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:10:49.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:10:49.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:49.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7f833c006a90 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:49.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7f833c006a90 tx=0x7f833c0512e0).stop 2021-03-09T09:10:49.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:49.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:49.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:49.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:49.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:49.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f834c0f4690 2021-03-09T09:10:49.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012b40 con 0x7f834c0f6280 2021-03-09T09:10:49.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eba0 con 0x7f8338062ca0 2021-03-09T09:10:49.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:10:49.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:49.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:10:49.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:49.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:49.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:49.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:49.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:49.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:49.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380662e0 con 0x7f834c0f6280 2021-03-09T09:10:49.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807e300 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7f833c04d4a0 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:49.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c029ee0 con 0x7f834c0f6280 2021-03-09T09:10:49.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c005800 con 0x7f834c0f6280 2021-03-09T09:10:49.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:49.467+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c078c70 con 0x7f834c0f6280 2021-03-09T09:10:49.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:10:49 smithi146 bash[13136]: debug 2021-03-09T09:10:49.456+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2430 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:10:54.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:10:54.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:54.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807e300 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7f833c04d4a0 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:54.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807e300 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7f833c04d4a0 tx=0x7f833c0512e0).stop 2021-03-09T09:10:54.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807e300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:54.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807e300 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:54.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:54.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:54.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:54.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380675f0 con 0x7f834c0f4690 2021-03-09T09:10:54.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f8338006a70 2021-03-09T09:10:54.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004ba0 con 0x7f8338062ca0 2021-03-09T09:10:54.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:10:54.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:10:54.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:54.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:54.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:54.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:54.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:54.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:54.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:54.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.459+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801eae0 con 0x7f8338006a70 2021-03-09T09:10:54.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.463+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834405a000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:54.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029040 con 0x7f8338006a70 2021-03-09T09:10:54.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834401f120 con 0x7f8338006a70 2021-03-09T09:10:54.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:54.467+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344051420 con 0x7f8338006a70 2021-03-09T09:10:54.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:10:54 smithi146 bash[13136]: debug 2021-03-09T09:10:54.460+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2457 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:10:59.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.458+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:10:59.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.458+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:59.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.458+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834405a000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:59.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.458+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834405a000).stop 2021-03-09T09:10:59.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.458+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:59.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.458+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:59.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.458+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:59.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.458+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:10:59.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.458+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:59.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.458+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:59.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.458+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:59.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.458+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ed50 con 0x7f834c0f4690 2021-03-09T09:10:59.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.458+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e090 con 0x7f834c0f6280 2021-03-09T09:10:59.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.458+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023d40 con 0x7f8338062ca0 2021-03-09T09:10:59.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.458+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:10:59.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.458+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:59.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.462+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:59.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.462+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:59.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.462+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:59.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.462+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:59.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.462+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:59.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.462+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338018a60 con 0x7f834c0f6280 2021-03-09T09:10:59.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.462+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015230 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f833c025620 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:59.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c021760 con 0x7f834c0f6280 2021-03-09T09:10:59.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c021760 con 0x7f834c0f6280 2021-03-09T09:10:59.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:10:59.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004180 con 0x7f834c0f6280 2021-03-09T09:10:59.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:10:59 smithi146 bash[13136]: debug 2021-03-09T09:10:59.460+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2481 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:11:04.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.458+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338015230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:11:04.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.458+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338015230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:04.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.458+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015230 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f833c025620 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:04.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.458+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015230 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f833c025620 tx=0x7f833c0512e0).stop 2021-03-09T09:11:04.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.458+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338015230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:04.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.458+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338015230 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:04.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.458+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:04.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.458+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:04.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.458+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:04.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.458+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a10 con 0x7f834c0f4690 2021-03-09T09:11:04.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.458+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c020 con 0x7f8338006a70 2021-03-09T09:11:04.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.458+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380653f0 con 0x7f8338062ca0 2021-03-09T09:11:04.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.458+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:11:04.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.458+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:04.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.462+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:04.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.462+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:11:04.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.462+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:04.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.462+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:04.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.462+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:04.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.462+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:04.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.462+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:04.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.462+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338018a60 con 0x7f8338006a70 2021-03-09T09:11:04.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.462+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0x7f833400a1b0 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:04.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f8338006a70 2021-03-09T09:11:04.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833400bb90 con 0x7f8338006a70 2021-03-09T09:11:04.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:04.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340020d0 con 0x7f8338006a70 2021-03-09T09:11:04.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:11:04 smithi146 bash[13136]: debug 2021-03-09T09:11:04.460+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2509 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:11:09.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:11:09.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:09.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0x7f833400a1b0 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:09.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0x7f833400a1b0 tx=0x7f8334049000).stop 2021-03-09T09:11:09.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:09.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:09.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:09.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:09.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:09.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012b40 con 0x7f834c0f4690 2021-03-09T09:11:09.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f834c0f6280 2021-03-09T09:11:09.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338070090 con 0x7f8338062ca0 2021-03-09T09:11:09.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:11:09.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:09.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:11:09.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:09.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:09.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:09.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:09.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:09.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:09.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012360 con 0x7f834c0f6280 2021-03-09T09:11:09.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.462+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7f833c0568f0 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:09.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c056e00 con 0x7f834c0f6280 2021-03-09T09:11:09.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0193f0 con 0x7f834c0f6280 2021-03-09T09:11:09.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:09.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004070 con 0x7f834c0f6280 2021-03-09T09:11:09.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:11:09 smithi146 bash[13136]: debug 2021-03-09T09:11:09.460+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2531 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:11:14.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338007d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:11:14.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338007d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:14.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7f833c0568f0 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:14.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7f833c0568f0 tx=0x7f833c0512e0).stop 2021-03-09T09:11:14.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338007d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:14.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:14.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:14.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:14.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:14.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e730 con 0x7f834c0f4690 2021-03-09T09:11:14.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380183d0 con 0x7f8338006a70 2021-03-09T09:11:14.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a10 con 0x7f8338062ca0 2021-03-09T09:11:14.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:11:14.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:14.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:11:14.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:14.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:14.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:14.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:14.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:14.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:14.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.462+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024f60 con 0x7f8338006a70 2021-03-09T09:11:14.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.466+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404ac70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:14.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029040 con 0x7f8338006a70 2021-03-09T09:11:14.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404c6c0 con 0x7f8338006a70 2021-03-09T09:11:14.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:14.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344051420 con 0x7f8338006a70 2021-03-09T09:11:14.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:11:14 smithi146 bash[13136]: debug 2021-03-09T09:11:14.464+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2557 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:11:19.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:11:19.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:19.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404ac70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:19.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404ac70).stop 2021-03-09T09:11:19.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:19.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:19.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:19.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338028040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:19.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:19.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:11:19.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:19.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380653f0 con 0x7f834c0f4690 2021-03-09T09:11:19.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f834c0f6280 2021-03-09T09:11:19.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ad70 con 0x7f8338062ca0 2021-03-09T09:11:19.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:11:19.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:19.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.462+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338028040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:19.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.466+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:19.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.466+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:19.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.466+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:19.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.466+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:19.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.466+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064fc0 con 0x7f834c0f6280 2021-03-09T09:11:19.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.466+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338028040 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834404ac70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:19.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029040 con 0x7f834c0f6280 2021-03-09T09:11:19.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072610 con 0x7f834c0f6280 2021-03-09T09:11:19.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:19.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344047b40 con 0x7f834c0f6280 2021-03-09T09:11:19.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:11:19 smithi146 bash[13136]: debug 2021-03-09T09:11:19.464+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2581 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:11:24.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.462+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338028040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:11:24.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.462+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338028040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:24.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.462+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338028040 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834404ac70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:24.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.462+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338028040 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834404ac70).stop 2021-03-09T09:11:24.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338028040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:24.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.462+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338028040 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:24.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.462+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:24.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.462+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:24.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.462+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:24.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007bb0 con 0x7f834c0f4690 2021-03-09T09:11:24.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a10 con 0x7f8338006a70 2021-03-09T09:11:24.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062080 con 0x7f8338062ca0 2021-03-09T09:11:24.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.466+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:11:24.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.466+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:11:24.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.466+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:24.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.466+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:24.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.466+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:24.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.466+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:24.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.466+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:24.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.466+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:24.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.466+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:24.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.466+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380137f0 con 0x7f8338006a70 2021-03-09T09:11:24.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.466+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f833405fc40 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:24.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f8338006a70 2021-03-09T09:11:24.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007af0 con 0x7f8338006a70 2021-03-09T09:11:24.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:24.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340081f0 con 0x7f8338006a70 2021-03-09T09:11:24.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:11:24 smithi146 bash[13136]: debug 2021-03-09T09:11:24.468+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2607 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:11:29.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:11:29.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:29.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f833405fc40 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:29.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f833405fc40 tx=0x7f8334049000).stop 2021-03-09T09:11:29.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:29.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:29.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380114d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:29.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:29.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:29.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380653f0 con 0x7f834c0f4690 2021-03-09T09:11:29.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016960 con 0x7f834c0f6280 2021-03-09T09:11:29.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800abc0 con 0x7f8338062ca0 2021-03-09T09:11:29.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:11:29.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:29.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:11:29.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:29.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380114d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:29.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:29.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:29.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:29.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:29.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007d80 con 0x7f834c0f6280 2021-03-09T09:11:29.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.466+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380114d0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7f833404a7e0 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:29.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334029670 con 0x7f834c0f6280 2021-03-09T09:11:29.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334029490 con 0x7f834c0f6280 2021-03-09T09:11:29.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:29.474+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334017130 con 0x7f834c0f6280 2021-03-09T09:11:29.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:11:29 smithi146 bash[13136]: debug 2021-03-09T09:11:29.467+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2634 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:11:34.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:11:34.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:34.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380114d0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7f833404a7e0 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:34.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380114d0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7f833404a7e0 tx=0x7f8334049000).stop 2021-03-09T09:11:34.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380114d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:34.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380114d0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:34.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:34.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:34.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:34.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e730 con 0x7f834c0f4690 2021-03-09T09:11:34.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f9c0 con 0x7f8338006a70 2021-03-09T09:11:34.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011f00 con 0x7f8338062ca0 2021-03-09T09:11:34.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:11:34.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:34.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:34.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:11:34.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:34.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:34.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:34.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:34.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:34.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.466+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065f70 con 0x7f8338006a70 2021-03-09T09:11:34.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.470+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404ac70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:34.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f8338006a70 2021-03-09T09:11:34.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402ac20 con 0x7f8338006a70 2021-03-09T09:11:34.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:34.474+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834401f2a0 con 0x7f8338006a70 2021-03-09T09:11:34.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:11:34 smithi146 bash[13136]: debug 2021-03-09T09:11:34.467+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2656 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:11:39.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.466+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:11:39.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.466+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:39.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.466+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404ac70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:39.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.466+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404ac70).stop 2021-03-09T09:11:39.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:39.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.466+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:39.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.466+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:39.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.466+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:39.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.466+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:11:39.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.470+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:39.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.466+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:39.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.474+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f834c0f4690 2021-03-09T09:11:39.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.474+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380653f0 con 0x7f834c0f6280 2021-03-09T09:11:39.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.474+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027510 con 0x7f8338062ca0 2021-03-09T09:11:39.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.474+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:39.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.474+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:11:39.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.474+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:39.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.474+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:39.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.474+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:39.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.474+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:39.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.474+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:39.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.474+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f834c0f6280 2021-03-09T09:11:39.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.474+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806fcf0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f833405fc40 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:39.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.478+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334063180 con 0x7f834c0f6280 2021-03-09T09:11:39.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.478+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334022730 con 0x7f834c0f6280 2021-03-09T09:11:39.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:39.478+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f834c0f6280 2021-03-09T09:11:39.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:11:39 smithi146 bash[13136]: debug 2021-03-09T09:11:39.467+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2684 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:11:44.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:11:44.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:44.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806fcf0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f833405fc40 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:44.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806fcf0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f833405fc40 tx=0x7f8334049000).stop 2021-03-09T09:11:44.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806fcf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:44.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806fcf0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:44.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:44.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:44.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:44.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:11:44.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:44.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:11:44.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:44.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ab0 con 0x7f834c0f4690 2021-03-09T09:11:44.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016960 con 0x7f8338006a70 2021-03-09T09:11:44.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cd20 con 0x7f8338062ca0 2021-03-09T09:11:44.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:44.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:44.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:44.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:44.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:44.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a740 con 0x7f8338006a70 2021-03-09T09:11:44.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.470+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f833c05ef70 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:44.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.474+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068040 con 0x7f8338006a70 2021-03-09T09:11:44.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.474+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05ed70 con 0x7f8338006a70 2021-03-09T09:11:44.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:44.478+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338006a70 2021-03-09T09:11:44.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:11:44 smithi146 bash[13136]: debug 2021-03-09T09:11:44.471+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2709 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:11:49.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:11:49.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:49.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f833c05ef70 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:49.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f833c05ef70 tx=0x7f833c0512e0).stop 2021-03-09T09:11:49.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:49.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:49.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:49.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:49.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:49.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:11:49.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:49.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f834c0f4690 2021-03-09T09:11:49.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f3b0 con 0x7f834c0f6280 2021-03-09T09:11:49.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064740 con 0x7f8338062ca0 2021-03-09T09:11:49.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:11:49.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:49.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.470+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:49.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.474+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:49.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.474+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:49.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.474+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:49.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.474+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:49.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.474+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338061530 con 0x7f834c0f6280 2021-03-09T09:11:49.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.474+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404ac70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:49.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.478+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f834c0f6280 2021-03-09T09:11:49.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.478+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344016120 con 0x7f834c0f6280 2021-03-09T09:11:49.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:49.478+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344072230 con 0x7f834c0f6280 2021-03-09T09:11:49.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:11:49 smithi146 bash[13136]: debug 2021-03-09T09:11:49.471+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2731 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:11:54.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.469+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338007d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:11:54.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.469+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338007d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:54.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.469+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404ac70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:54.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.469+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404ac70).stop 2021-03-09T09:11:54.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338007d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:54.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.469+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007d80 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:54.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.469+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:54.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.469+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:54.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.469+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:54.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800abc0 con 0x7f834c0f4690 2021-03-09T09:11:54.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e920 con 0x7f8338006a70 2021-03-09T09:11:54.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011f00 con 0x7f8338062ca0 2021-03-09T09:11:54.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.469+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:11:54.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.469+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:11:54.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.469+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:54.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.473+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:54.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:54.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.473+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:54.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.473+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:54.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.473+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:54.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.473+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:54.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.473+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380030c0 con 0x7f8338006a70 2021-03-09T09:11:54.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.473+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7f8334070050 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:54.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e020 con 0x7f8338006a70 2021-03-09T09:11:54.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334063c20 con 0x7f8338006a70 2021-03-09T09:11:54.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:54.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334016940 con 0x7f8338006a70 2021-03-09T09:11:54.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:11:54 smithi146 bash[13136]: debug 2021-03-09T09:11:54.471+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2756 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:11:59.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.469+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:11:59.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.469+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:59.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.469+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7f8334070050 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:59.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.469+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7f8334070050 tx=0x7f8334049000).stop 2021-03-09T09:11:59.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:59.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.469+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:59.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.469+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:59.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.469+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:59.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.469+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380086f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:59.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380653f0 con 0x7f834c0f4690 2021-03-09T09:11:59.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f3b0 con 0x7f834c0f6280 2021-03-09T09:11:59.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ab0 con 0x7f8338062ca0 2021-03-09T09:11:59.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.469+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:11:59.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.469+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:59.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.469+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:11:59.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.469+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:59.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:59.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:59.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:59.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:59.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:59.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380030c0 con 0x7f834c0f6280 2021-03-09T09:11:59.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380086f0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:59.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068040 con 0x7f834c0f6280 2021-03-09T09:11:59.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c078e50 con 0x7f834c0f6280 2021-03-09T09:11:59.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:11:59.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f834c0f6280 2021-03-09T09:11:59.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:11:59 smithi146 bash[13136]: debug 2021-03-09T09:11:59.471+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2784 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:12:04.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.469+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:12:04.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.469+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:04.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.469+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380086f0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:04.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.469+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380086f0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c0512e0).stop 2021-03-09T09:12:04.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380086f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:04.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380086f0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:04.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:04.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:04.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:04.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e920 con 0x7f834c0f4690 2021-03-09T09:12:04.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f8338006a70 2021-03-09T09:12:04.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020d70 con 0x7f8338062ca0 2021-03-09T09:12:04.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:12:04.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:04.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:12:04.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:04.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:04.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:04.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:04.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:04.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:04.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ccd0 con 0x7f8338006a70 2021-03-09T09:12:04.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.477+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7f833c04e4f0 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:04.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068040 con 0x7f8338006a70 2021-03-09T09:12:04.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c070c30 con 0x7f8338006a70 2021-03-09T09:12:04.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:04.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c019210 con 0x7f8338006a70 2021-03-09T09:12:04.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:12:04 smithi146 bash[13136]: debug 2021-03-09T09:12:04.475+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2805 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:12:09.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:12:09.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:09.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7f833c04e4f0 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:09.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7f833c04e4f0 tx=0x7f833c0512e0).stop 2021-03-09T09:12:09.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:09.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:09.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:09.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:09.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:09.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380185d0 con 0x7f834c0f4690 2021-03-09T09:12:09.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ab0 con 0x7f834c0f6280 2021-03-09T09:12:09.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f8338062ca0 2021-03-09T09:12:09.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:12:09.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:09.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:09.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:12:09.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:09.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:09.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:09.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:09.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:09.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.473+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062960 con 0x7f834c0f6280 2021-03-09T09:12:09.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.477+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009c80 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7f8334007920 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:09.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e030 con 0x7f834c0f6280 2021-03-09T09:12:09.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334022070 con 0x7f834c0f6280 2021-03-09T09:12:09.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:09.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833400b980 con 0x7f834c0f6280 2021-03-09T09:12:09.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:12:09 smithi146 bash[13136]: debug 2021-03-09T09:12:09.475+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2830 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:12:14.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:12:14.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:14.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009c80 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7f8334007920 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:14.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009c80 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7f8334007920 tx=0x7f8334049000).stop 2021-03-09T09:12:14.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:14.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009c80 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:14.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:14.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:14.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:14.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016b60 con 0x7f834c0f4690 2021-03-09T09:12:14.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e920 con 0x7f8338006a70 2021-03-09T09:12:14.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f8338062ca0 2021-03-09T09:12:14.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:12:14.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:12:14.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:14.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:14.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:14.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:14.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:14.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:14.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:14.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338025a20 con 0x7f8338006a70 2021-03-09T09:12:14.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0x7f833800b100 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:14.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344016120 con 0x7f8338006a70 2021-03-09T09:12:14.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405c270 con 0x7f8338006a70 2021-03-09T09:12:14.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:14.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344048020 con 0x7f8338006a70 2021-03-09T09:12:14.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:12:14 smithi146 bash[13136]: debug 2021-03-09T09:12:14.475+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2856 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:12:19.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:12:19.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:19.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0x7f833800b100 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:19.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0x7f833800b100 tx=0x7f8344004650).stop 2021-03-09T09:12:19.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:19.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:19.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:19.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:19.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:19.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006890 con 0x7f834c0f4690 2021-03-09T09:12:19.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380167c0 con 0x7f834c0f6280 2021-03-09T09:12:19.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071310 con 0x7f8338062ca0 2021-03-09T09:12:19.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:12:19.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:19.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:19.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:12:19.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:19.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:19.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:19.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:19.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:19.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a170 con 0x7f834c0f6280 2021-03-09T09:12:19.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009c80 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:19.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f834c0f6280 2021-03-09T09:12:19.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f834c0f6280 2021-03-09T09:12:19.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:19.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02a100 con 0x7f834c0f6280 2021-03-09T09:12:19.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:12:19 smithi146 bash[13136]: debug 2021-03-09T09:12:19.475+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2881 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:12:24.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.477+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:12:24.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.477+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:24.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.477+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009c80 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:24.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.477+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009c80 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c0512e0).stop 2021-03-09T09:12:24.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:24.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009c80 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:24.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:24.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:24.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:24.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ab0 con 0x7f834c0f4690 2021-03-09T09:12:24.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f8338006a70 2021-03-09T09:12:24.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016b60 con 0x7f8338062ca0 2021-03-09T09:12:24.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.477+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:12:24.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.477+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:12:24.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.477+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:24.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.477+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:24.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.481+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:24.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.485+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:24.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.485+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:24.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.485+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:24.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.485+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:24.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.485+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ab70 con 0x7f8338006a70 2021-03-09T09:12:24.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.485+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7f834405c4c0 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:24.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338006a70 2021-03-09T09:12:24.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344049800 con 0x7f8338006a70 2021-03-09T09:12:24.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:24.489+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338006a70 2021-03-09T09:12:24.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:12:24 smithi146 bash[13136]: debug 2021-03-09T09:12:24.479+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2905 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:12:29.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:12:29.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:29.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7f834405c4c0 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:29.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7f834405c4c0 tx=0x7f8344004650).stop 2021-03-09T09:12:29.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:29.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:29.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:29.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:29.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:29.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ab0 con 0x7f834c0f4690 2021-03-09T09:12:29.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dad0 con 0x7f834c0f6280 2021-03-09T09:12:29.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006890 con 0x7f8338062ca0 2021-03-09T09:12:29.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:12:29.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:29.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:12:29.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:29.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.477+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:29.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.481+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:29.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.481+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:29.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.481+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:29.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.481+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:29.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.481+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380157b0 con 0x7f834c0f6280 2021-03-09T09:12:29.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.481+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025e80 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:29.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f834c0f6280 2021-03-09T09:12:29.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007920 con 0x7f834c0f6280 2021-03-09T09:12:29.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:29.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340085d0 con 0x7f834c0f6280 2021-03-09T09:12:29.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:12:29 smithi146 bash[13136]: debug 2021-03-09T09:12:29.479+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2930 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:12:34.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:12:34.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:34.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025e80 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:34.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025e80 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f8334049000).stop 2021-03-09T09:12:34.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:34.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025e80 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:34.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:34.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:34.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:34.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cd20 con 0x7f834c0f4690 2021-03-09T09:12:34.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380167c0 con 0x7f8338006a70 2021-03-09T09:12:34.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380653f0 con 0x7f8338062ca0 2021-03-09T09:12:34.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:12:34.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:34.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:12:34.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:34.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:34.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:34.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:34.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:34.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:34.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068ef0 con 0x7f8338006a70 2021-03-09T09:12:34.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:34.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f8338006a70 2021-03-09T09:12:34.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334003140 con 0x7f8338006a70 2021-03-09T09:12:34.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:34.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334044b60 con 0x7f8338006a70 2021-03-09T09:12:34.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:12:34 smithi146 bash[13136]: debug 2021-03-09T09:12:34.479+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2957 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:12:39.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:12:39.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:39.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:39.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f8334049000).stop 2021-03-09T09:12:39.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:39.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:39.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:39.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:39.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:39.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066690 con 0x7f834c0f4690 2021-03-09T09:12:39.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f834c0f6280 2021-03-09T09:12:39.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ab0 con 0x7f8338062ca0 2021-03-09T09:12:39.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:12:39.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:39.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:12:39.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:39.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:39.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:39.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:39.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:39.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:39.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806da80 con 0x7f834c0f6280 2021-03-09T09:12:39.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.481+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7f833c0566f0 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:39.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f834c0f6280 2021-03-09T09:12:39.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f834c0f6280 2021-03-09T09:12:39.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:39.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c056c60 con 0x7f834c0f6280 2021-03-09T09:12:39.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:12:39 smithi146 bash[13136]: debug 2021-03-09T09:12:39.483+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 2981 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:12:44.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:12:44.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:44.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7f833c0566f0 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:44.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7f833c0566f0 tx=0x7f833c0512e0).stop 2021-03-09T09:12:44.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338067ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:44.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:44.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:44.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:44.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:44.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f9c0 con 0x7f834c0f4690 2021-03-09T09:12:44.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380658c0 con 0x7f8338006a70 2021-03-09T09:12:44.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cd20 con 0x7f8338062ca0 2021-03-09T09:12:44.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:12:44.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:44.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:12:44.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:44.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:44.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:44.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:44.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:44.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:44.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.481+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ab00 con 0x7f8338006a70 2021-03-09T09:12:44.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.485+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7f8334007e00 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:44.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f8338006a70 2021-03-09T09:12:44.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334017130 con 0x7f8338006a70 2021-03-09T09:12:44.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:44.489+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338006a70 2021-03-09T09:12:44.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:12:44 smithi146 bash[13136]: debug 2021-03-09T09:12:44.482+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3006 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:12:49.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:12:49.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:49.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7f8334007e00 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:49.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7f8334007e00 tx=0x7f8334049000).stop 2021-03-09T09:12:49.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:49.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:49.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:49.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:49.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:49.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ab0 con 0x7f834c0f4690 2021-03-09T09:12:49.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a0d0 con 0x7f834c0f6280 2021-03-09T09:12:49.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071310 con 0x7f8338062ca0 2021-03-09T09:12:49.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:12:49.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:12:49.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:49.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:49.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.480+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:49.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.484+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:49.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.484+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:49.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.484+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:49.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.484+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:49.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.484+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338006840 con 0x7f834c0f6280 2021-03-09T09:12:49.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.484+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:49.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f834c0f6280 2021-03-09T09:12:49.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f834c0f6280 2021-03-09T09:12:49.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:49.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f834c0f6280 2021-03-09T09:12:49.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:12:49 smithi146 bash[13136]: debug 2021-03-09T09:12:49.482+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3033 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:12:54.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:12:54.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:54.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:54.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c0512e0).stop 2021-03-09T09:12:54.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338067ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:54.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338067ee0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:54.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:54.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:54.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:54.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f9c0 con 0x7f834c0f4690 2021-03-09T09:12:54.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806f680 con 0x7f8338006a70 2021-03-09T09:12:54.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cd20 con 0x7f8338062ca0 2021-03-09T09:12:54.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:12:54.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:54.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:12:54.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:54.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:54.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:54.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:54.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:54.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:54.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338011eb0 con 0x7f8338006a70 2021-03-09T09:12:54.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.484+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0x7f8334026bc0 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:54.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032030 con 0x7f8338006a70 2021-03-09T09:12:54.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340181d0 con 0x7f8338006a70 2021-03-09T09:12:54.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:54.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334016a20 con 0x7f8338006a70 2021-03-09T09:12:54.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:12:54 smithi146 bash[13136]: debug 2021-03-09T09:12:54.482+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3055 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:12:59.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:12:59.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:59.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0x7f8334026bc0 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:59.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0x7f8334026bc0 tx=0x7f8334049000).stop 2021-03-09T09:12:59.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:59.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:59.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:59.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:59.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338002b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:59.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:12:59.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ef00 con 0x7f834c0f4690 2021-03-09T09:12:59.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f834c0f6280 2021-03-09T09:12:59.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d840 con 0x7f8338062ca0 2021-03-09T09:12:59.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:59.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:12:59.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:59.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338002b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:59.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:59.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:59.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:59.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:59.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.484+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014590 con 0x7f834c0f6280 2021-03-09T09:12:59.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338002b30 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7f833c007010 tx=0x7f833c046040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:59.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f834c0f6280 2021-03-09T09:12:59.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0552f0 con 0x7f834c0f6280 2021-03-09T09:12:59.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:12:59.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f834c0f6280 2021-03-09T09:12:59.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:12:59 smithi146 bash[13136]: debug 2021-03-09T09:12:59.486+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3081 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:13:04.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338002b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:13:04.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338002b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:04.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338002b30 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7f833c007010 tx=0x7f833c046040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:04.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338002b30 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7f833c007010 tx=0x7f833c046040).stop 2021-03-09T09:13:04.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338002b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:04.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338002b30 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:04.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:04.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:04.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:04.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806daa0 con 0x7f834c0f4690 2021-03-09T09:13:04.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010260 con 0x7f8338006a70 2021-03-09T09:13:04.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f9c0 con 0x7f8338062ca0 2021-03-09T09:13:04.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:13:04.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:04.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:13:04.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:04.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.484+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:04.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:04.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:04.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:04.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:04.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b120 con 0x7f8338006a70 2021-03-09T09:13:04.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7f833c053750 tx=0x7f833c046040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:04.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068030 con 0x7f8338006a70 2021-03-09T09:13:04.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c049600 con 0x7f8338006a70 2021-03-09T09:13:04.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:04.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05e1b0 con 0x7f8338006a70 2021-03-09T09:13:04.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:13:04 smithi146 bash[13136]: debug 2021-03-09T09:13:04.486+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3107 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:13:09.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:13:09.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:09.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7f833c053750 tx=0x7f833c046040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:09.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7f833c053750 tx=0x7f833c046040).stop 2021-03-09T09:13:09.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:09.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:09.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:09.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:09.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:13:09.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:09.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:09.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bbc0 con 0x7f834c0f4690 2021-03-09T09:13:09.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f834c0f6280 2021-03-09T09:13:09.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806daa0 con 0x7f8338062ca0 2021-03-09T09:13:09.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:13:09.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:09.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:09.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:09.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:09.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:09.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:09.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801ad30 con 0x7f834c0f6280 2021-03-09T09:13:09.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.492+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380064e0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c046040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:09.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f834c0f6280 2021-03-09T09:13:09.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c066140 con 0x7f834c0f6280 2021-03-09T09:13:09.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:09.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0055f0 con 0x7f834c0f6280 2021-03-09T09:13:09.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:13:09 smithi146 bash[13136]: debug 2021-03-09T09:13:09.486+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3132 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:13:14.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:13:14.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:14.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380064e0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c046040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:14.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380064e0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c046040).stop 2021-03-09T09:13:14.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380064e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:14.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380064e0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:14.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:14.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:14.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:14.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ce40 con 0x7f834c0f4690 2021-03-09T09:13:14.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004e80 con 0x7f8338006a70 2021-03-09T09:13:14.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f8338062ca0 2021-03-09T09:13:14.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:13:14.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:14.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:13:14.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:14.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:14.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:14.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:14.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:14.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:14.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338061530 con 0x7f8338006a70 2021-03-09T09:13:14.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0x7f8334009bd0 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:14.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032030 con 0x7f8338006a70 2021-03-09T09:13:14.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340261b0 con 0x7f8338006a70 2021-03-09T09:13:14.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:14.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833400b980 con 0x7f8338006a70 2021-03-09T09:13:14.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:13:14 smithi146 bash[13136]: debug 2021-03-09T09:13:14.490+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3157 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:13:19.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:13:19.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:19.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0x7f8334009bd0 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:19.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0x7f8334009bd0 tx=0x7f8334049000).stop 2021-03-09T09:13:19.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:19.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:19.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:19.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:13:19.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:19.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:19.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:13:19.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:19.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:19.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023bc0 con 0x7f834c0f4690 2021-03-09T09:13:19.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380288d0 con 0x7f834c0f6280 2021-03-09T09:13:19.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f8338062ca0 2021-03-09T09:13:19.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:19.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:19.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:19.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:19.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:19.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017410 con 0x7f834c0f6280 2021-03-09T09:13:19.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.492+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c046040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:19.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f834c0f6280 2021-03-09T09:13:19.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c071890 con 0x7f834c0f6280 2021-03-09T09:13:19.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:19.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04de90 con 0x7f834c0f6280 2021-03-09T09:13:19.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:13:19 smithi146 bash[13136]: debug 2021-03-09T09:13:19.490+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3183 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:13:24.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338069990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:13:24.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338069990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:24.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c046040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:24.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c046040).stop 2021-03-09T09:13:24.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338069990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:24.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:24.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:24.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:24.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:24.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:24.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f9c0 con 0x7f834c0f4690 2021-03-09T09:13:24.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007fc0 con 0x7f8338006a70 2021-03-09T09:13:24.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8338062ca0 2021-03-09T09:13:24.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:13:24.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:24.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:13:24.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:24.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:24.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:24.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:24.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:24.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800eeb0 con 0x7f8338006a70 2021-03-09T09:13:24.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0x7f833406b1e0 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:24.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f8338006a70 2021-03-09T09:13:24.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334024340 con 0x7f8338006a70 2021-03-09T09:13:24.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:24.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833400b980 con 0x7f8338006a70 2021-03-09T09:13:24.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:13:24 smithi146 bash[13136]: debug 2021-03-09T09:13:24.490+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3207 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:13:29.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:13:29.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:29.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0x7f833406b1e0 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:29.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0x7f833406b1e0 tx=0x7f8334049000).stop 2021-03-09T09:13:29.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:29.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:29.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:29.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:29.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:29.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023bc0 con 0x7f834c0f4690 2021-03-09T09:13:29.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380024c0 con 0x7f834c0f6280 2021-03-09T09:13:29.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065b00 con 0x7f8338062ca0 2021-03-09T09:13:29.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:13:29.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:29.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:13:29.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:29.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:29.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:29.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.492+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:29.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.496+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:29.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.496+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:29.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.496+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380167c0 con 0x7f834c0f6280 2021-03-09T09:13:29.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.496+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7f8338008590 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:29.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f834c0f6280 2021-03-09T09:13:29.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f834c0f6280 2021-03-09T09:13:29.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:29.500+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023070 con 0x7f834c0f6280 2021-03-09T09:13:29.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:13:29 smithi146 bash[13136]: debug 2021-03-09T09:13:29.494+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3230 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:13:34.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338069990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:13:34.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338069990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:34.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7f8338008590 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:34.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7f8338008590 tx=0x7f8344004650).stop 2021-03-09T09:13:34.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338069990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:34.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:34.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:34.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:34.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:34.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:13:34.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:34.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:13:34.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:34.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:34.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800def0 con 0x7f834c0f4690 2021-03-09T09:13:34.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bd50 con 0x7f8338006a70 2021-03-09T09:13:34.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380158a0 con 0x7f8338062ca0 2021-03-09T09:13:34.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:34.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:34.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:34.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:34.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.492+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338063e20 con 0x7f8338006a70 2021-03-09T09:13:34.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.496+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:34.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401ed70 con 0x7f8338006a70 2021-03-09T09:13:34.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344050850 con 0x7f8338006a70 2021-03-09T09:13:34.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:34.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344048500 con 0x7f8338006a70 2021-03-09T09:13:34.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:13:34 smithi146 bash[13136]: debug 2021-03-09T09:13:34.494+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3259 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:13:39.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:13:39.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:39.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:39.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f8344004650).stop 2021-03-09T09:13:39.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:39.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:39.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:39.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:39.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:39.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063ce0 con 0x7f834c0f4690 2021-03-09T09:13:39.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:13:39.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d5a0 con 0x7f834c0f6280 2021-03-09T09:13:39.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f9c0 con 0x7f8338062ca0 2021-03-09T09:13:39.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:39.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:13:39.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:39.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:39.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:39.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:39.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:39.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:39.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338063e20 con 0x7f834c0f6280 2021-03-09T09:13:39.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.495+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007700 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7f8344004620 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:39.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344047440 con 0x7f834c0f6280 2021-03-09T09:13:39.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440475a0 con 0x7f834c0f6280 2021-03-09T09:13:39.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:39.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ac30 con 0x7f834c0f6280 2021-03-09T09:13:39.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:13:39 smithi146 bash[13136]: debug 2021-03-09T09:13:39.494+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3280 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:13:44.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338007700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:13:44.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338007700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:44.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007700 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7f8344004620 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:44.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007700 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7f8344004620 tx=0x7f8344004650).stop 2021-03-09T09:13:44.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338007700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:44.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338007700 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:44.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:44.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:44.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:44.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:13:44.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:44.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013b80 con 0x7f834c0f4690 2021-03-09T09:13:44.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bbc0 con 0x7f8338006a70 2021-03-09T09:13:44.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013390 con 0x7f8338062ca0 2021-03-09T09:13:44.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:13:44.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:44.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.495+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:44.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.499+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:44.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.499+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:44.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.499+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:44.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.499+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:44.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.499+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028880 con 0x7f8338006a70 2021-03-09T09:13:44.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.499+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0x7f8334046010 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:44.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f8338006a70 2021-03-09T09:13:44.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334056e50 con 0x7f8338006a70 2021-03-09T09:13:44.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:44.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334018b50 con 0x7f8338006a70 2021-03-09T09:13:44.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:13:44 smithi146 bash[13136]: debug 2021-03-09T09:13:44.498+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3305 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:13:49.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.495+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:13:49.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.495+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:49.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.495+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0x7f8334046010 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:49.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.495+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0x7f8334046010 tx=0x7f8334049000).stop 2021-03-09T09:13:49.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:49.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:49.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338011490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:49.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:49.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:49.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b360 con 0x7f834c0f4690 2021-03-09T09:13:49.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800def0 con 0x7f834c0f6280 2021-03-09T09:13:49.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dd30 con 0x7f8338062ca0 2021-03-09T09:13:49.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:13:49.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:49.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338011490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:49.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:13:49.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:49.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:49.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:49.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:49.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:49.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a5d0 con 0x7f834c0f6280 2021-03-09T09:13:49.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338011490 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f83340176b0 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:49.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334007c60 con 0x7f834c0f6280 2021-03-09T09:13:49.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334018910 con 0x7f834c0f6280 2021-03-09T09:13:49.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:49.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833400d340 con 0x7f834c0f6280 2021-03-09T09:13:49.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:13:49 smithi146 bash[13136]: debug 2021-03-09T09:13:49.498+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3332 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:13:54.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338011490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:13:54.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338011490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:54.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338011490 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f83340176b0 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:54.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338011490 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f83340176b0 tx=0x7f8334049000).stop 2021-03-09T09:13:54.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338011490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:54.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338011490 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:54.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:54.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:54.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:54.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013b80 con 0x7f834c0f4690 2021-03-09T09:13:54.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f8338006a70 2021-03-09T09:13:54.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380182e0 con 0x7f8338062ca0 2021-03-09T09:13:54.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:13:54.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:54.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:13:54.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:54.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.499+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:54.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.503+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:54.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.503+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:54.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.503+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:54.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.503+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:54.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.503+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800fc00 con 0x7f8338006a70 2021-03-09T09:13:54.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.503+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0x7f8338008620 tx=0x7f833c071ba0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:54.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068040 con 0x7f8338006a70 2021-03-09T09:13:54.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05eb30 con 0x7f8338006a70 2021-03-09T09:13:54.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:54.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338006a70 2021-03-09T09:13:54.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:13:54 smithi146 bash[13136]: debug 2021-03-09T09:13:54.498+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3358 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:13:59.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:13:59.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:59.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0x7f8338008620 tx=0x7f833c071ba0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:59.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0x7f8338008620 tx=0x7f833c071ba0).stop 2021-03-09T09:13:59.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:59.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:59.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:59.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:59.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338062350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:59.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380024c0 con 0x7f834c0f4690 2021-03-09T09:13:59.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f834c0f6280 2021-03-09T09:13:59.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800def0 con 0x7f8338062ca0 2021-03-09T09:13:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:13:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:13:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.499+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338062350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.503+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.503+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.503+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.503+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:59.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.503+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062840 con 0x7f834c0f6280 2021-03-09T09:13:59.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.503+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338062350 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0x7f833c02a800 tx=0x7f833c009250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:59.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f834c0f6280 2021-03-09T09:13:59.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c02ade0 con 0x7f834c0f6280 2021-03-09T09:13:59.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:13:59.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04a530 con 0x7f834c0f6280 2021-03-09T09:13:59.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:13:59 smithi146 bash[13136]: debug 2021-03-09T09:13:59.498+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3378 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:14:04.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.499+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338062350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:14:04.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.499+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338062350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:04.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.499+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338062350 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0x7f833c02a800 tx=0x7f833c009250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:04.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.499+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338062350 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0x7f833c02a800 tx=0x7f833c009250).stop 2021-03-09T09:14:04.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338062350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:04.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338062350 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:04.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:04.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:04.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:04.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005e80 con 0x7f834c0f4690 2021-03-09T09:14:04.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013b80 con 0x7f8338006a70 2021-03-09T09:14:04.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f8338062ca0 2021-03-09T09:14:04.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.503+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:04.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.503+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:14:04.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.503+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:04.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.503+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:14:04.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.503+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:04.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.503+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:04.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.503+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:04.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.503+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:04.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.503+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:04.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.503+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800cdf0 con 0x7f8338006a70 2021-03-09T09:14:04.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.503+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0x7f8338002a00 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:04.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f8338006a70 2021-03-09T09:14:04.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402a900 con 0x7f8338006a70 2021-03-09T09:14:04.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:04.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023040 con 0x7f8338006a70 2021-03-09T09:14:04.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:14:04 smithi146 bash[13136]: debug 2021-03-09T09:14:04.501+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3406 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:14:09.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:14:09.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:09.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0x7f8338002a00 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:09.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0x7f8338002a00 tx=0x7f8344004650).stop 2021-03-09T09:14:09.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:09.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:09.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:09.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:09.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064070 con 0x7f834c0f4690 2021-03-09T09:14:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802db30 con 0x7f834c0f6280 2021-03-09T09:14:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:14:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380182e0 con 0x7f8338062ca0 2021-03-09T09:14:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:09.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:14:09.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:09.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.503+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:09.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.507+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:09.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.507+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:09.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.507+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:09.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.507+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:09.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.507+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338019330 con 0x7f834c0f6280 2021-03-09T09:14:09.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.507+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7f8334000ea0 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:09.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f834c0f6280 2021-03-09T09:14:09.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f834c0f6280 2021-03-09T09:14:09.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:09.511+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833400b980 con 0x7f834c0f6280 2021-03-09T09:14:09.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:14:09 smithi146 bash[13136]: debug 2021-03-09T09:14:09.501+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3428 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:14:14.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338069990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:14:14.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338069990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:14.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7f8334000ea0 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:14.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7f8334000ea0 tx=0x7f8334049000).stop 2021-03-09T09:14:14.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338069990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:14.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069990 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:14.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:14.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:14.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:14.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380696a0 con 0x7f834c0f4690 2021-03-09T09:14:14.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061ce0 con 0x7f8338006a70 2021-03-09T09:14:14.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028ad0 con 0x7f8338062ca0 2021-03-09T09:14:14.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:14:14.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:14.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:14:14.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:14.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.503+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:14.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.507+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:14.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.507+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:14.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.507+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:14.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.507+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:14.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.507+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338019330 con 0x7f8338006a70 2021-03-09T09:14:14.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.507+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:14.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e9c0 con 0x7f8338006a70 2021-03-09T09:14:14.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007c60 con 0x7f8338006a70 2021-03-09T09:14:14.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:14.511+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833400e650 con 0x7f8338006a70 2021-03-09T09:14:14.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:14:14 smithi146 bash[13136]: debug 2021-03-09T09:14:14.501+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3456 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:14:19.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:14:19.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:19.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:19.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334049000).stop 2021-03-09T09:14:19.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:19.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:19.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:19.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:19.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:19.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:14:19.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:19.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:14:19.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:19.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f834c0f4690 2021-03-09T09:14:19.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023bc0 con 0x7f834c0f6280 2021-03-09T09:14:19.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802db30 con 0x7f8338062ca0 2021-03-09T09:14:19.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.503+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:19.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.507+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:19.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.507+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:19.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.507+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:19.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.507+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:19.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.507+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338061530 con 0x7f834c0f6280 2021-03-09T09:14:19.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.507+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069940 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:19.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334019a10 con 0x7f834c0f6280 2021-03-09T09:14:19.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334019630 con 0x7f834c0f6280 2021-03-09T09:14:19.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:19.511+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334003c20 con 0x7f834c0f6280 2021-03-09T09:14:19.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:14:19 smithi146 bash[13136]: debug 2021-03-09T09:14:19.505+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3480 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:14:24.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338069940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:14:24.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338069940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:24.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069940 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:24.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069940 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334049000).stop 2021-03-09T09:14:24.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338069940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:24.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338069940 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:24.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:24.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:24.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:14:24.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:14:24.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:24.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:24.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:24.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064070 con 0x7f834c0f4690 2021-03-09T09:14:24.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f8338006a70 2021-03-09T09:14:24.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380182e0 con 0x7f8338062ca0 2021-03-09T09:14:24.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:24.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:24.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:24.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:24.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:24.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.507+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338019330 con 0x7f8338006a70 2021-03-09T09:14:24.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.511+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0x7f833c009f90 tx=0x7f833c04d460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:24.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f8338006a70 2021-03-09T09:14:24.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c078220 con 0x7f8338006a70 2021-03-09T09:14:24.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:24.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04de90 con 0x7f8338006a70 2021-03-09T09:14:24.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:14:24 smithi146 bash[13136]: debug 2021-03-09T09:14:24.505+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3505 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:14:29.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:14:29.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:29.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0x7f833c009f90 tx=0x7f833c04d460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:29.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0x7f833c009f90 tx=0x7f833c04d460).stop 2021-03-09T09:14:29.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:29.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:29.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:29.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:29.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:14:29.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:29.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:14:29.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:29.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380708d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:29.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023bc0 con 0x7f834c0f4690 2021-03-09T09:14:29.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005e80 con 0x7f834c0f6280 2021-03-09T09:14:29.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013b80 con 0x7f8338062ca0 2021-03-09T09:14:29.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.507+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:29.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.511+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:29.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.511+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:29.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.511+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:29.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.511+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:29.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.511+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802cb70 con 0x7f834c0f6280 2021-03-09T09:14:29.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.511+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380708d0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04d460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:29.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.511+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c02a9b0 con 0x7f834c0f6280 2021-03-09T09:14:29.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.511+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f834c0f6280 2021-03-09T09:14:29.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:29.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02a100 con 0x7f834c0f6280 2021-03-09T09:14:29.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:14:29 smithi146 bash[13136]: debug 2021-03-09T09:14:29.505+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3529 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:14:34.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:14:34.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:34.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380708d0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04d460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:34.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380708d0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04d460).stop 2021-03-09T09:14:34.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380708d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:34.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380708d0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:34.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:34.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:34.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:14:34.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:34.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:34.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064070 con 0x7f834c0f4690 2021-03-09T09:14:34.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a6b0 con 0x7f8338006a70 2021-03-09T09:14:34.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061ce0 con 0x7f8338062ca0 2021-03-09T09:14:34.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:14:34.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:34.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.506+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:34.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.510+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:34.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:34.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.510+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:34.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:34.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.510+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338021d10 con 0x7f8338006a70 2021-03-09T09:14:34.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:34.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032030 con 0x7f8338006a70 2021-03-09T09:14:34.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334047ec0 con 0x7f8338006a70 2021-03-09T09:14:34.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:34.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334033b00 con 0x7f8338006a70 2021-03-09T09:14:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:14:34 smithi146 bash[13136]: debug 2021-03-09T09:14:34.505+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3559 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:14:39.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:14:39.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:39.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:39.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334049000).stop 2021-03-09T09:14:39.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:39.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:39.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:39.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:39.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:14:39.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:39.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:39.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380658f0 con 0x7f834c0f4690 2021-03-09T09:14:39.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380147e0 con 0x7f834c0f6280 2021-03-09T09:14:39.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f8338062ca0 2021-03-09T09:14:39.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:14:39.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:39.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:39.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:39.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:39.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:39.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:39.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338019330 con 0x7f834c0f6280 2021-03-09T09:14:39.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009da0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:39.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f834c0f6280 2021-03-09T09:14:39.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334019390 con 0x7f834c0f6280 2021-03-09T09:14:39.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:39.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401f950 con 0x7f834c0f6280 2021-03-09T09:14:40.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:14:39 smithi146 bash[13136]: debug 2021-03-09T09:14:39.509+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3578 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:14:44.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:14:44.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:44.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009da0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:44.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009da0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334049000).stop 2021-03-09T09:14:44.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:44.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009da0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:44.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:44.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:44.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:44.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f834c0f4690 2021-03-09T09:14:44.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380026c0 con 0x7f8338006a70 2021-03-09T09:14:44.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011690 con 0x7f8338062ca0 2021-03-09T09:14:44.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:14:44.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:14:44.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:44.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:44.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:44.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:44.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:44.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:44.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:44.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022de0 con 0x7f8338006a70 2021-03-09T09:14:44.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.510+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:44.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401f950 con 0x7f8338006a70 2021-03-09T09:14:44.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f8338006a70 2021-03-09T09:14:44.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:44.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023070 con 0x7f8338006a70 2021-03-09T09:14:45.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:14:44 smithi146 bash[13136]: debug 2021-03-09T09:14:44.509+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3609 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:14:49.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:14:49.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:49.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:49.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f8344004650).stop 2021-03-09T09:14:49.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:49.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:49.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:49.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:49.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:49.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023dc0 con 0x7f834c0f4690 2021-03-09T09:14:49.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380147e0 con 0x7f834c0f6280 2021-03-09T09:14:49.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f8338062ca0 2021-03-09T09:14:49.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:14:49.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:49.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:14:49.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:49.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:49.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.514+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:49.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.514+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:49.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.514+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:49.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.514+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:49.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.514+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338021d10 con 0x7f834c0f6280 2021-03-09T09:14:49.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.514+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:49.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f834c0f6280 2021-03-09T09:14:49.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f834c0f6280 2021-03-09T09:14:49.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:49.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023030 con 0x7f834c0f6280 2021-03-09T09:14:50.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:14:49 smithi146 bash[13136]: debug 2021-03-09T09:14:49.513+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3632 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:14:50.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:50.634+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833805c1c0 msgr2=0x7f833805e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900003568) for more than 900000000 us, fault. 2021-03-09T09:14:50.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:50.634+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833805c1c0 0x7f833805e680 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f834c0ff600 tx=0x7f833c006ac0).stop 2021-03-09T09:14:50.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:50.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833805c1c0 msgr2=0x7f833805e680 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:50.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:50.634+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833805c1c0 0x7f833805e680 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:50.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:50.634+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:50.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:50.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8334007af0 con 0x7f8338006a70 2021-03-09T09:14:50.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:50.634+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:50.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:50.634+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c039000).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:54.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.510+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:14:54.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.510+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:54.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.510+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:54.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.510+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344004650).stop 2021-03-09T09:14:54.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806c690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:54.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:54.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:54.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:54.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:54.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d750 con 0x7f834c0f4690 2021-03-09T09:14:54.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026960 con 0x7f833805c1c0 2021-03-09T09:14:54.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065df0 con 0x7f8338062ca0 2021-03-09T09:14:54.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.510+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:14:54.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.510+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:54.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:54.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.514+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:14:54.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.514+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:54.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:54.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:54.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:54.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:54.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338019330 con 0x7f833805c1c0 2021-03-09T09:14:54.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5e0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0x7f8334019760 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:54.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f833805c1c0 2021-03-09T09:14:54.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334061210 con 0x7f833805c1c0 2021-03-09T09:14:54.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:54.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404b1a0 con 0x7f833805c1c0 2021-03-09T09:14:55.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:14:54 smithi146 bash[13136]: debug 2021-03-09T09:14:54.513+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3654 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:14:59.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:14:59.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:59.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5e0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0x7f8334019760 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:59.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5e0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0x7f8334019760 tx=0x7f8334049000).stop 2021-03-09T09:14:59.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:59.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5e0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:59.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:59.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:59.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:59.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f834c0f4690 2021-03-09T09:14:59.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f9c0 con 0x7f834c0f6280 2021-03-09T09:14:59.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380026c0 con 0x7f8338062ca0 2021-03-09T09:14:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:14:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:14:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338004e80 con 0x7f834c0f6280 2021-03-09T09:14:59.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:59.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f834c0f6280 2021-03-09T09:14:59.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833400e820 con 0x7f834c0f6280 2021-03-09T09:14:59.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:14:59.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334052c60 con 0x7f834c0f6280 2021-03-09T09:15:00.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:14:59 smithi146 bash[13136]: debug 2021-03-09T09:14:59.513+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3678 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:15:04.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:15:04.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:04.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:04.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334049000).stop 2021-03-09T09:15:04.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806c690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:04.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:04.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:04.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:04.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:15:04.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:04.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:04.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f834c0f4690 2021-03-09T09:15:04.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d7a0 con 0x7f833805c1c0 2021-03-09T09:15:04.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d7a0 con 0x7f8338062ca0 2021-03-09T09:15:04.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:15:04.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:04.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.514+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:04.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.518+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:04.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:04.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.518+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:04.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:04.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.518+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020670 con 0x7f833805c1c0 2021-03-09T09:15:04.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:04.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f833805c1c0 2021-03-09T09:15:04.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f833805c1c0 2021-03-09T09:15:04.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:04.522+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344023030 con 0x7f833805c1c0 2021-03-09T09:15:05.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:15:04 smithi146 bash[13136]: debug 2021-03-09T09:15:04.517+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3706 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:15:09.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.514+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:15:09.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.514+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:09.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.514+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:09.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.514+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004650).stop 2021-03-09T09:15:09.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:09.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:09.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:09.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:09.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:09.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071310 con 0x7f834c0f4690 2021-03-09T09:15:09.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ef00 con 0x7f834c0f6280 2021-03-09T09:15:09.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023dc0 con 0x7f8338062ca0 2021-03-09T09:15:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:15:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.514+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:15:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.514+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.518+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.518+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:09.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.518+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022de0 con 0x7f834c0f6280 2021-03-09T09:15:09.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7f8344017a40 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:09.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f834c0f6280 2021-03-09T09:15:09.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072190 con 0x7f834c0f6280 2021-03-09T09:15:09.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:09.522+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440171c0 con 0x7f834c0f6280 2021-03-09T09:15:10.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:15:09 smithi146 bash[13136]: debug 2021-03-09T09:15:09.513+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3729 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:15:14.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.514+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:15:14.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.514+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:14.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.514+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7f8344017a40 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:14.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.514+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7f8344017a40 tx=0x7f8344004650).stop 2021-03-09T09:15:14.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806c690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:14.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806c690 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:14.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:14.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:14.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:14.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f834c0f4690 2021-03-09T09:15:14.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071310 con 0x7f833805c1c0 2021-03-09T09:15:14.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380700f0 con 0x7f8338062ca0 2021-03-09T09:15:14.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:15:14.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:14.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:15:14.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:14.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:14.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:14.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:14.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:14.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:14.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338061530 con 0x7f833805c1c0 2021-03-09T09:15:14.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0x7f83440164c0 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:14.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.522+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344027e70 con 0x7f833805c1c0 2021-03-09T09:15:14.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.522+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f833805c1c0 2021-03-09T09:15:14.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:14.526+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400b610 con 0x7f833805c1c0 2021-03-09T09:15:15.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:15:14 smithi146 bash[13136]: debug 2021-03-09T09:15:14.517+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3757 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:15:19.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:15:19.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:19.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0x7f83440164c0 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:19.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0x7f83440164c0 tx=0x7f8344004650).stop 2021-03-09T09:15:19.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:19.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:19.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:19.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:19.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338065a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:19.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014680 con 0x7f834c0f4690 2021-03-09T09:15:19.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f834c0f6280 2021-03-09T09:15:19.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022270 con 0x7f8338062ca0 2021-03-09T09:15:19.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:15:19.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:19.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:15:19.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338065a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:19.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:19.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.522+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:19.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.522+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:19.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.522+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:19.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.522+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:19.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.522+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338023dc0 con 0x7f834c0f6280 2021-03-09T09:15:19.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.522+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338065a80 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c03a000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:19.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.522+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068040 con 0x7f834c0f6280 2021-03-09T09:15:19.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.522+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007af0 con 0x7f834c0f6280 2021-03-09T09:15:19.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:19.526+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f834c0f6280 2021-03-09T09:15:20.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:15:19 smithi146 bash[13136]: debug 2021-03-09T09:15:19.521+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3781 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:15:24.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.518+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338065a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:15:24.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.518+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338065a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:24.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.518+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338065a80 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c03a000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:24.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.518+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338065a80 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c03a000).stop 2021-03-09T09:15:24.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338065a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:24.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338065a80 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:24.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:24.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.518+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:15:24.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.518+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:24.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:24.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:24.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006080 con 0x7f834c0f4690 2021-03-09T09:15:24.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d7a0 con 0x7f833805c1c0 2021-03-09T09:15:24.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f8338062ca0 2021-03-09T09:15:24.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.518+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:15:24.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.518+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:24.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.522+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:24.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.522+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:24.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.522+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:24.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.522+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:24.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.522+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:24.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.522+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f833805c1c0 2021-03-09T09:15:24.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.522+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:24.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.522+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344002f50 con 0x7f833805c1c0 2021-03-09T09:15:24.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.522+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344020170 con 0x7f833805c1c0 2021-03-09T09:15:24.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:24.526+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404e110 con 0x7f833805c1c0 2021-03-09T09:15:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:15:24 smithi146 bash[13136]: debug 2021-03-09T09:15:24.521+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3808 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:15:29.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:15:29.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:29.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:29.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344004650).stop 2021-03-09T09:15:29.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:29.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:29.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:29.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:15:29.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:29.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:29.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:29.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d7d0 con 0x7f834c0f4690 2021-03-09T09:15:29.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067250 con 0x7f834c0f6280 2021-03-09T09:15:29.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f8338062ca0 2021-03-09T09:15:29.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:15:29.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:29.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:29.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:29.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:29.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:29.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:29.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.521+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338023dc0 con 0x7f834c0f6280 2021-03-09T09:15:29.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.525+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0x7f8338026cd0 tx=0x7f833c03a000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:29.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068040 con 0x7f834c0f6280 2021-03-09T09:15:29.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c071e40 con 0x7f834c0f6280 2021-03-09T09:15:29.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:29.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f834c0f6280 2021-03-09T09:15:30.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:15:29 smithi146 bash[13136]: debug 2021-03-09T09:15:29.520+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3833 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:15:34.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:15:34.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:34.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0x7f8338026cd0 tx=0x7f833c03a000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:34.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0x7f8338026cd0 tx=0x7f833c03a000).stop 2021-03-09T09:15:34.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338019650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:34.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:34.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:34.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:34.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:34.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028c80 con 0x7f834c0f4690 2021-03-09T09:15:34.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006080 con 0x7f833805c1c0 2021-03-09T09:15:34.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f9c0 con 0x7f8338062ca0 2021-03-09T09:15:34.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:15:34.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:34.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:15:34.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:34.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.521+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:34.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.525+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:34.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.525+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:34.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.525+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:34.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.525+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:34.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.525+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014630 con 0x7f833805c1c0 2021-03-09T09:15:34.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.525+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7f833c05ffc0 tx=0x7f833c03a000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:34.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f833805c1c0 2021-03-09T09:15:34.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833805c1c0 2021-03-09T09:15:34.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:34.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833805c1c0 2021-03-09T09:15:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:15:34 smithi146 bash[13136]: debug 2021-03-09T09:15:34.520+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3857 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:15:39.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.521+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:15:39.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.521+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:39.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.521+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7f833c05ffc0 tx=0x7f833c03a000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:39.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.521+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7f833c05ffc0 tx=0x7f833c03a000).stop 2021-03-09T09:15:39.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.521+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:39.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.521+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:39.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.521+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338068da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:39.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.521+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:39.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.521+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:39.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.521+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022270 con 0x7f834c0f4690 2021-03-09T09:15:39.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.521+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f834c0f6280 2021-03-09T09:15:39.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.521+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028c80 con 0x7f8338062ca0 2021-03-09T09:15:39.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.521+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:15:39.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.521+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:39.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.521+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338068da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:39.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.525+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:15:39.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.525+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:39.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.529+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:39.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.529+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:39.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.529+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:39.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.529+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:39.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.529+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338023dc0 con 0x7f834c0f6280 2021-03-09T09:15:39.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.529+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338068da0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0x7f8334023d00 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:39.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340191b0 con 0x7f834c0f6280 2021-03-09T09:15:39.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334019310 con 0x7f834c0f6280 2021-03-09T09:15:39.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:39.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403a1f0 con 0x7f834c0f6280 2021-03-09T09:15:40.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:15:39 smithi146 bash[13136]: debug 2021-03-09T09:15:39.524+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3883 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:15:44.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338068da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:15:44.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338068da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:44.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338068da0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0x7f8334023d00 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:44.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338068da0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0x7f8334023d00 tx=0x7f8334049000).stop 2021-03-09T09:15:44.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338068da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:44.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338068da0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:44.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:44.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:15:44.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:44.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:44.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:44.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e100 con 0x7f834c0f4690 2021-03-09T09:15:44.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006080 con 0x7f833805c1c0 2021-03-09T09:15:44.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028c80 con 0x7f8338062ca0 2021-03-09T09:15:44.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:15:44.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:44.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:44.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:44.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:44.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:44.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:44.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380696e0 con 0x7f833805c1c0 2021-03-09T09:15:44.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.525+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7f8338031770 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:44.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f833805c1c0 2021-03-09T09:15:44.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072190 con 0x7f833805c1c0 2021-03-09T09:15:44.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:44.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404ad20 con 0x7f833805c1c0 2021-03-09T09:15:45.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:15:44 smithi146 bash[13136]: debug 2021-03-09T09:15:44.524+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3905 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:15:49.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:15:49.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7f8338031770 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7f8338031770 tx=0x7f8344004650).stop 2021-03-09T09:15:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:49.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:49.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:49.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:49.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f834c0f4690 2021-03-09T09:15:49.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006930 con 0x7f834c0f6280 2021-03-09T09:15:49.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f8338062ca0 2021-03-09T09:15:49.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:15:49.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:49.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:49.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:15:49.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:49.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:49.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:49.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:49.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:49.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.525+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064040 con 0x7f834c0f6280 2021-03-09T09:15:49.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.529+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7f833c056910 tx=0x7f833c03a000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:49.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f834c0f6280 2021-03-09T09:15:49.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f834c0f6280 2021-03-09T09:15:49.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:49.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f834c0f6280 2021-03-09T09:15:50.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:15:49 smithi146 bash[13136]: debug 2021-03-09T09:15:49.524+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3931 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:15:54.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:15:54.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:54.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7f833c056910 tx=0x7f833c03a000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:54.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7f833c056910 tx=0x7f833c03a000).stop 2021-03-09T09:15:54.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338019650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:54.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:54.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:54.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:54.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:54.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e100 con 0x7f834c0f4690 2021-03-09T09:15:54.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f9c0 con 0x7f833805c1c0 2021-03-09T09:15:54.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006080 con 0x7f8338062ca0 2021-03-09T09:15:54.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:15:54.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:54.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:15:54.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:54.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.525+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:54.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.529+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:54.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.529+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:54.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.529+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:54.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.529+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:54.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.529+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005a60 con 0x7f833805c1c0 2021-03-09T09:15:54.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.529+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:54.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f833805c1c0 2021-03-09T09:15:54.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007960 con 0x7f833805c1c0 2021-03-09T09:15:54.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:54.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d220 con 0x7f833805c1c0 2021-03-09T09:15:55.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:15:54 smithi146 bash[13136]: debug 2021-03-09T09:15:54.528+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3957 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:15:59.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.525+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:15:59.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.525+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:59.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.525+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:59.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.525+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f8334049000).stop 2021-03-09T09:15:59.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:59.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:59.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:59.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:59.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:59.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006080 con 0x7f834c0f4690 2021-03-09T09:15:59.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f834c0f6280 2021-03-09T09:15:59.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e100 con 0x7f8338062ca0 2021-03-09T09:15:59.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.525+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:15:59.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.529+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:15:59.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.529+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:59.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.525+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:59.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.529+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:59.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.529+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:59.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.529+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:59.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.529+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.529+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.529+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800cff0 con 0x7f834c0f6280 2021-03-09T09:15:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.529+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7f833c029fc0 tx=0x7f833c03a000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:59.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c035370 con 0x7f834c0f6280 2021-03-09T09:15:59.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f834c0f6280 2021-03-09T09:15:59.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:15:59.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f834c0f6280 2021-03-09T09:16:00.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:15:59 smithi146 bash[13136]: debug 2021-03-09T09:15:59.524+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 3981 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:16:04.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:16:04.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:04.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7f833c029fc0 tx=0x7f833c03a000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:04.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7f833c029fc0 tx=0x7f833c03a000).stop 2021-03-09T09:16:04.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338019650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:04.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338019650 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:04.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:04.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:04.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:04.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e100 con 0x7f834c0f4690 2021-03-09T09:16:04.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006080 con 0x7f833805c1c0 2021-03-09T09:16:04.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027b30 con 0x7f8338062ca0 2021-03-09T09:16:04.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:16:04.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:16:04.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:04.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:04.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:04.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a5e0 con 0x7f833805c1c0 2021-03-09T09:16:04.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.529+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801a650 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:04.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f833805c1c0 2021-03-09T09:16:04.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402db10 con 0x7f833805c1c0 2021-03-09T09:16:04.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:04.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334052a50 con 0x7f833805c1c0 2021-03-09T09:16:05.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:16:04 smithi146 bash[13136]: debug 2021-03-09T09:16:04.528+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4007 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:16:09.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:16:09.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:09.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801a650 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:09.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801a650 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334049000).stop 2021-03-09T09:16:09.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801a650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:09.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801a650 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:09.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e100 con 0x7f834c0f4690 2021-03-09T09:16:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338070740 con 0x7f834c0f6280 2021-03-09T09:16:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833805eb50 con 0x7f8338062ca0 2021-03-09T09:16:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:16:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:16:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:09.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.529+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:09.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.533+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:09.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.533+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:09.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.533+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:09.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.533+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:09.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.533+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028c30 con 0x7f834c0f6280 2021-03-09T09:16:09.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.537+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806d080 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c03a000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:09.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.537+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f834c0f6280 2021-03-09T09:16:09.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.537+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f834c0f6280 2021-03-09T09:16:09.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:09.541+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f834c0f6280 2021-03-09T09:16:10.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:16:09 smithi146 bash[13136]: debug 2021-03-09T09:16:09.532+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4029 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:16:14.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:16:14.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:14.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806d080 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c03a000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:14.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806d080 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c03a000).stop 2021-03-09T09:16:14.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833806d080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:14.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833806d080 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:14.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:14.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:14.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:14.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b10 con 0x7f834c0f4690 2021-03-09T09:16:14.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006080 con 0x7f833805c1c0 2021-03-09T09:16:14.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e100 con 0x7f8338062ca0 2021-03-09T09:16:14.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:16:14.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:14.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:16:14.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:14.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:14.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:14.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:14.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:14.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:14.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800cff0 con 0x7f833805c1c0 2021-03-09T09:16:14.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7f8338002ac0 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:14.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f833805c1c0 2021-03-09T09:16:14.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402b760 con 0x7f833805c1c0 2021-03-09T09:16:14.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:14.537+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833805c1c0 2021-03-09T09:16:15.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:16:14 smithi146 bash[13136]: debug 2021-03-09T09:16:14.532+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4058 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:16:19.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:16:19.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:19.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7f8338002ac0 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:19.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7f8338002ac0 tx=0x7f8344004650).stop 2021-03-09T09:16:19.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:19.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:19.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338070600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:19.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:19.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:19.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f834c0f4690 2021-03-09T09:16:19.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f834c0f6280 2021-03-09T09:16:19.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e100 con 0x7f8338062ca0 2021-03-09T09:16:19.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:16:19.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:19.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:16:19.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:19.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338070600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:19.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:19.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:19.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:19.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:19.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.533+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028a50 con 0x7f834c0f6280 2021-03-09T09:16:19.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.537+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338070600 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0x7f8334060560 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:19.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.537+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f834c0f6280 2021-03-09T09:16:19.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.537+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334060360 con 0x7f834c0f6280 2021-03-09T09:16:19.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:19.541+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334052450 con 0x7f834c0f6280 2021-03-09T09:16:20.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:16:19 smithi146 bash[13136]: debug 2021-03-09T09:16:19.532+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4083 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:16:24.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338070600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:16:24.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338070600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:24.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338070600 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0x7f8334060560 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:24.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338070600 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0x7f8334060560 tx=0x7f8334049000).stop 2021-03-09T09:16:24.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338070600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:24.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338070600 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:24.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:24.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:24.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:24.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800da00 con 0x7f834c0f4690 2021-03-09T09:16:24.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026740 con 0x7f833805c1c0 2021-03-09T09:16:24.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b380 con 0x7f8338062ca0 2021-03-09T09:16:24.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:16:24.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:24.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:16:24.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:24.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:24.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:24.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:24.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:24.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:24.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338061530 con 0x7f833805c1c0 2021-03-09T09:16:24.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7f833c019ce0 tx=0x7f833c03a000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:24.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068040 con 0x7f833805c1c0 2021-03-09T09:16:24.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.536+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833805c1c0 2021-03-09T09:16:24.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:24.540+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04de90 con 0x7f833805c1c0 2021-03-09T09:16:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:16:24 smithi146 bash[13136]: debug 2021-03-09T09:16:24.532+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4106 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:16:29.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.532+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:16:29.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.532+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:29.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.532+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7f833c019ce0 tx=0x7f833c03a000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:29.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.532+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7f833c019ce0 tx=0x7f833c03a000).stop 2021-03-09T09:16:29.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.532+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:29.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.532+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:29.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.532+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:29.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.532+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:29.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.532+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:29.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.532+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f9c0 con 0x7f834c0f4690 2021-03-09T09:16:29.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.532+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806ca70 con 0x7f834c0f6280 2021-03-09T09:16:29.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.532+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006080 con 0x7f8338062ca0 2021-03-09T09:16:29.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.532+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:16:29.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.532+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:29.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.532+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:16:29.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.536+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:29.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.536+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:29.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.536+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:29.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.536+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:29.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.536+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:29.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.536+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:29.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.536+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c190 con 0x7f834c0f6280 2021-03-09T09:16:29.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.536+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d6a0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c03a000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:29.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.536+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068040 con 0x7f834c0f6280 2021-03-09T09:16:29.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.536+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c060da0 con 0x7f834c0f6280 2021-03-09T09:16:29.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:29.540+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c020c90 con 0x7f834c0f6280 2021-03-09T09:16:30.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:16:29 smithi146 bash[13136]: debug 2021-03-09T09:16:29.536+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4132 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:16:34.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:16:34.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:34.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d6a0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c03a000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:34.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d6a0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c03a000).stop 2021-03-09T09:16:34.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802d6a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:34.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d6a0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:34.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:34.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:16:34.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:34.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:34.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:34.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b10 con 0x7f834c0f4690 2021-03-09T09:16:34.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e100 con 0x7f833805c1c0 2021-03-09T09:16:34.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8338062ca0 2021-03-09T09:16:34.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:16:34.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:34.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.536+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:34.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.540+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:34.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.540+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:34.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.540+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:34.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.540+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:34.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.540+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028c30 con 0x7f833805c1c0 2021-03-09T09:16:34.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.540+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 rx=0x7f833c053e90 tx=0x7f833c03a000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:34.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.540+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068040 con 0x7f833805c1c0 2021-03-09T09:16:34.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.540+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0538c0 con 0x7f833805c1c0 2021-03-09T09:16:34.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:34.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c029d10 con 0x7f833805c1c0 2021-03-09T09:16:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:16:34 smithi146 bash[13136]: debug 2021-03-09T09:16:34.540+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4157 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:16:39.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:16:39.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:39.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 rx=0x7f833c053e90 tx=0x7f833c03a000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:39.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 rx=0x7f833c053e90 tx=0x7f833c03a000).stop 2021-03-09T09:16:39.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:39.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:39.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:39.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:39.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:39.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f9c0 con 0x7f834c0f4690 2021-03-09T09:16:39.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d640 con 0x7f834c0f6280 2021-03-09T09:16:39.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f8338062ca0 2021-03-09T09:16:39.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:16:39.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:39.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:39.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:16:39.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.536+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:39.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.540+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:39.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.540+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:39.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.540+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:39.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.540+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:39.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.540+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c190 con 0x7f834c0f6280 2021-03-09T09:16:39.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.540+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7f833400be90 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:39.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032080 con 0x7f834c0f6280 2021-03-09T09:16:39.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833405f170 con 0x7f834c0f6280 2021-03-09T09:16:39.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:39.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334052c60 con 0x7f834c0f6280 2021-03-09T09:16:40.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:16:39 smithi146 bash[13136]: debug 2021-03-09T09:16:39.539+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4178 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:16:44.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:16:44.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:44.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7f833400be90 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:44.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7f833400be90 tx=0x7f8334049000).stop 2021-03-09T09:16:44.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:44.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:44.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:44.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:44.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:44.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002870 con 0x7f834c0f4690 2021-03-09T09:16:44.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806ca70 con 0x7f833805c1c0 2021-03-09T09:16:44.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ede0 con 0x7f8338062ca0 2021-03-09T09:16:44.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:16:44.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:44.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:16:44.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:44.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:44.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:44.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:44.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:44.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:44.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014a10 con 0x7f833805c1c0 2021-03-09T09:16:44.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.540+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 rx=0x7f8338067320 tx=0x7f833c008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:44.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f833805c1c0 2021-03-09T09:16:44.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c078bf0 con 0x7f833805c1c0 2021-03-09T09:16:44.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:44.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00d280 con 0x7f833805c1c0 2021-03-09T09:16:45.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:16:44 smithi146 bash[13136]: debug 2021-03-09T09:16:44.539+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4207 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:16:49.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:16:49.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:49.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 rx=0x7f8338067320 tx=0x7f833c008040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:49.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 rx=0x7f8338067320 tx=0x7f833c008040).stop 2021-03-09T09:16:49.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:49.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:49.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:49.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338004780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:49.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:16:49.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:49.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:49.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a260 con 0x7f834c0f4690 2021-03-09T09:16:49.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dfa0 con 0x7f834c0f6280 2021-03-09T09:16:49.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027350 con 0x7f8338062ca0 2021-03-09T09:16:49.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:16:49.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:49.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:49.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:49.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:49.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:49.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:49.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.540+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380111c0 con 0x7f834c0f6280 2021-03-09T09:16:49.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.544+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338004780 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:49.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f834c0f6280 2021-03-09T09:16:49.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834401f950 con 0x7f834c0f6280 2021-03-09T09:16:49.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:49.552+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834401fd60 con 0x7f834c0f6280 2021-03-09T09:16:50.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:16:49 smithi146 bash[13136]: debug 2021-03-09T09:16:49.543+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4232 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:16:54.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:16:54.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:54.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338004780 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:54.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338004780 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344004650).stop 2021-03-09T09:16:54.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338004780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:54.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338004780 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:54.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:54.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:54.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:54.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380686d0 con 0x7f834c0f4690 2021-03-09T09:16:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ede0 con 0x7f833805c1c0 2021-03-09T09:16:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e100 con 0x7f8338062ca0 2021-03-09T09:16:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:16:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:16:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:54.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800d560 con 0x7f833805c1c0 2021-03-09T09:16:54.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 rx=0x7f833c01a430 tx=0x7f833c008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:54.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f833805c1c0 2021-03-09T09:16:54.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c035370 con 0x7f833805c1c0 2021-03-09T09:16:54.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:54.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c056210 con 0x7f833805c1c0 2021-03-09T09:16:55.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:16:54 smithi146 bash[13136]: debug 2021-03-09T09:16:54.543+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4258 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:16:59.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:16:59.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:59.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 rx=0x7f833c01a430 tx=0x7f833c008040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:59.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 rx=0x7f833c01a430 tx=0x7f833c008040).stop 2021-03-09T09:16:59.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:59.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:59.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:59.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:59.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:59.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ede0 con 0x7f834c0f4690 2021-03-09T09:16:59.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011210 con 0x7f834c0f6280 2021-03-09T09:16:59.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003080 con 0x7f8338062ca0 2021-03-09T09:16:59.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:16:59.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:59.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:16:59.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:59.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:59.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:59.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:59.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:59.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:59.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380136c0 con 0x7f834c0f6280 2021-03-09T09:16:59.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.544+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d6a0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f834406b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:59.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f834c0f6280 2021-03-09T09:16:59.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404ef00 con 0x7f834c0f6280 2021-03-09T09:16:59.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:16:59.552+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f834c0f6280 2021-03-09T09:17:00.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:16:59 smithi146 bash[13136]: debug 2021-03-09T09:16:59.543+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4279 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:17:04.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:17:04.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:04.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d6a0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f834406b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:04.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d6a0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f834406b040).stop 2021-03-09T09:17:04.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802d6a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:04.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802d6a0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:04.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:04.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:04.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:04.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380686d0 con 0x7f834c0f4690 2021-03-09T09:17:04.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066610 con 0x7f833805c1c0 2021-03-09T09:17:04.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380070b0 con 0x7f8338062ca0 2021-03-09T09:17:04.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:17:04.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:04.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:17:04.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:04.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:04.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:04.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:04.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:04.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:04.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338002820 con 0x7f833805c1c0 2021-03-09T09:17:04.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:04.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f833805c1c0 2021-03-09T09:17:04.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833400bb90 con 0x7f833805c1c0 2021-03-09T09:17:04.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:04.552+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402da60 con 0x7f833805c1c0 2021-03-09T09:17:05.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:17:04 smithi146 bash[13136]: debug 2021-03-09T09:17:04.547+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4308 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:17:09.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:17:09.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:09.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:09.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334021000).stop 2021-03-09T09:17:09.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:09.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:09.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:09.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:09.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:09.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ede0 con 0x7f834c0f4690 2021-03-09T09:17:09.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006930 con 0x7f834c0f6280 2021-03-09T09:17:09.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338070d20 con 0x7f8338062ca0 2021-03-09T09:17:09.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:17:09.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:17:09.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:09.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:09.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:09.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:09.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:09.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:09.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:09.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338018d50 con 0x7f834c0f6280 2021-03-09T09:17:09.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802c100 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 rx=0x7f833c04e4f0 tx=0x7f833c008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:09.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f834c0f6280 2021-03-09T09:17:09.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f080 con 0x7f834c0f6280 2021-03-09T09:17:09.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:09.552+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f834c0f6280 2021-03-09T09:17:10.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:17:09 smithi146 bash[13136]: debug 2021-03-09T09:17:09.547+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4331 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:17:14.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:17:14.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:14.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802c100 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 rx=0x7f833c04e4f0 tx=0x7f833c008040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:14.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802c100 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 rx=0x7f833c04e4f0 tx=0x7f833c008040).stop 2021-03-09T09:17:14.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802c100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:14.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802c100 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:14.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:14.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:14.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003080 con 0x7f834c0f4690 2021-03-09T09:17:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011210 con 0x7f833805c1c0 2021-03-09T09:17:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ede0 con 0x7f8338062ca0 2021-03-09T09:17:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:17:14.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:17:14.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:14.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:14.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:14.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:14.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:14.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338006030 con 0x7f833805c1c0 2021-03-09T09:17:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 rx=0x7f833c06ae50 tx=0x7f833c008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f833805c1c0 2021-03-09T09:17:14.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.548+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f030 con 0x7f833805c1c0 2021-03-09T09:17:14.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:14.552+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c020ba0 con 0x7f833805c1c0 2021-03-09T09:17:15.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:17:14 smithi146 bash[13136]: debug 2021-03-09T09:17:14.547+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4358 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:17:19.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:17:19.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:19.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 rx=0x7f833c06ae50 tx=0x7f833c008040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:19.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 rx=0x7f833c06ae50 tx=0x7f833c008040).stop 2021-03-09T09:17:19.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:19.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:19.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:19.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:19.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:17:19.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:17:19.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:19.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:19.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380248a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:19.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021c90 con 0x7f834c0f4690 2021-03-09T09:17:19.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f9c0 con 0x7f834c0f6280 2021-03-09T09:17:19.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f8338062ca0 2021-03-09T09:17:19.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380248a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:19.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:19.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:19.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:19.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:19.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068680 con 0x7f834c0f6280 2021-03-09T09:17:19.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380248a0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 rx=0x7f833c056580 tx=0x7f833c008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:19.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04de90 con 0x7f834c0f6280 2021-03-09T09:17:19.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.551+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04dff0 con 0x7f834c0f6280 2021-03-09T09:17:19.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:19.555+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04de90 con 0x7f834c0f6280 2021-03-09T09:17:20.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:17:19 smithi146 bash[13136]: debug 2021-03-09T09:17:19.551+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4381 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:17:24.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380248a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:17:24.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380248a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:24.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380248a0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 rx=0x7f833c056580 tx=0x7f833c008040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:24.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380248a0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 rx=0x7f833c056580 tx=0x7f833c008040).stop 2021-03-09T09:17:24.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380248a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:24.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380248a0 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:24.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:24.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:24.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:17:24.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:24.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:24.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ede0 con 0x7f834c0f4690 2021-03-09T09:17:24.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f833805c1c0 2021-03-09T09:17:24.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011210 con 0x7f8338062ca0 2021-03-09T09:17:24.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:17:24.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:24.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.551+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:24.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.555+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:24.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.555+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:24.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.555+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:24.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.555+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:24.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.555+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338018d50 con 0x7f833805c1c0 2021-03-09T09:17:24.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.555+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 rx=0x7f8338011c70 tx=0x7f834406b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:24.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404ef00 con 0x7f833805c1c0 2021-03-09T09:17:24.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405b2a0 con 0x7f833805c1c0 2021-03-09T09:17:24.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:24.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833805c1c0 2021-03-09T09:17:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:17:24 smithi146 bash[13136]: debug 2021-03-09T09:17:24.551+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4408 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:17:29.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.551+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:17:29.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.551+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:29.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.551+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 rx=0x7f8338011c70 tx=0x7f834406b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:29.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.551+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 rx=0x7f8338011c70 tx=0x7f834406b040).stop 2021-03-09T09:17:29.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:29.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:29.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:29.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:29.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:29.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f9c0 con 0x7f834c0f4690 2021-03-09T09:17:29.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f834c0f6280 2021-03-09T09:17:29.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066610 con 0x7f8338062ca0 2021-03-09T09:17:29.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:17:29.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:17:29.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:29.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:29.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:29.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:29.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:29.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:29.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:29.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.555+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800bec0 con 0x7f834c0f6280 2021-03-09T09:17:29.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.559+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccf0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 rx=0x7f833c01a310 tx=0x7f833c008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:29.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068040 con 0x7f834c0f6280 2021-03-09T09:17:29.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c053c00 con 0x7f834c0f6280 2021-03-09T09:17:29.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:29.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f834c0f6280 2021-03-09T09:17:30.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:17:29 smithi146 bash[13136]: debug 2021-03-09T09:17:29.551+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4431 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:17:34.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:17:34.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:34.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccf0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 rx=0x7f833c01a310 tx=0x7f833c008040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:34.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccf0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 rx=0x7f833c01a310 tx=0x7f833c008040).stop 2021-03-09T09:17:34.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802ccf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:34.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccf0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:34.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:34.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:34.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:17:34.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:34.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024f70 con 0x7f834c0f4690 2021-03-09T09:17:34.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:34.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068960 con 0x7f833805c1c0 2021-03-09T09:17:34.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ede0 con 0x7f8338062ca0 2021-03-09T09:17:34.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:17:34.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:34.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.555+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:34.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.559+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:34.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.559+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:34.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.559+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:34.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.559+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:34.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.559+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338066e10 con 0x7f833805c1c0 2021-03-09T09:17:34.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.559+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 rx=0x7f833c041af0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:34.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068040 con 0x7f833805c1c0 2021-03-09T09:17:34.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c060d10 con 0x7f833805c1c0 2021-03-09T09:17:34.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:34.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c056280 con 0x7f833805c1c0 2021-03-09T09:17:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:17:34 smithi146 bash[13136]: debug 2021-03-09T09:17:34.554+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4458 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:17:39.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.555+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:17:39.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.555+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:39.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.555+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 rx=0x7f833c041af0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:39.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.555+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 rx=0x7f833c041af0 tx=0x7f833c04e870).stop 2021-03-09T09:17:39.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.555+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:39.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.555+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:39.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.555+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:39.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.555+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:39.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.555+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:39.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.555+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:17:39.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.555+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:39.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.555+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:17:39.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.555+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:39.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.555+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016610 con 0x7f834c0f4690 2021-03-09T09:17:39.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.555+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027350 con 0x7f834c0f6280 2021-03-09T09:17:39.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.555+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006930 con 0x7f8338062ca0 2021-03-09T09:17:39.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.559+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:39.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.563+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:39.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.563+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:39.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.563+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:39.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.563+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:39.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.563+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800bec0 con 0x7f834c0f6280 2021-03-09T09:17:39.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.567+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025e80 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 rx=0x7f8334064fe0 tx=0x7f8334021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:39.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.567+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f834c0f6280 2021-03-09T09:17:39.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.567+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334052e60 con 0x7f834c0f6280 2021-03-09T09:17:39.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:39.567+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340173f0 con 0x7f834c0f6280 2021-03-09T09:17:40.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:17:39 smithi146 bash[13136]: debug 2021-03-09T09:17:39.554+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4479 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:17:44.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.555+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:17:44.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.555+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:44.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.555+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025e80 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 rx=0x7f8334064fe0 tx=0x7f8334021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:44.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.555+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025e80 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 rx=0x7f8334064fe0 tx=0x7f8334021000).stop 2021-03-09T09:17:44.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:44.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338025e80 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:44.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338026f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:44.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:44.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:44.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a410 con 0x7f834c0f4690 2021-03-09T09:17:44.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ede0 con 0x7f833805c1c0 2021-03-09T09:17:44.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f070 con 0x7f8338062ca0 2021-03-09T09:17:44.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:17:44.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:44.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:17:44.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:44.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338026f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:44.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:44.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:44.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:44.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:44.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.559+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338066e10 con 0x7f833805c1c0 2021-03-09T09:17:44.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.563+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338026f10 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 rx=0x7f833404bd30 tx=0x7f8334021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:44.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f833805c1c0 2021-03-09T09:17:44.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402dee0 con 0x7f833805c1c0 2021-03-09T09:17:44.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:44.567+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402da60 con 0x7f833805c1c0 2021-03-09T09:17:45.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:17:44 smithi146 bash[13136]: debug 2021-03-09T09:17:44.558+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4508 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:17:49.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338026f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:17:49.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338026f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:49.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338026f10 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 rx=0x7f833404bd30 tx=0x7f8334021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:49.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338026f10 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 rx=0x7f833404bd30 tx=0x7f8334021000).stop 2021-03-09T09:17:49.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338026f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:49.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338026f10 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:49.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:49.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:49.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338013950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:49.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005200 con 0x7f834c0f4690 2021-03-09T09:17:49.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006930 con 0x7f834c0f6280 2021-03-09T09:17:49.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f8338062ca0 2021-03-09T09:17:49.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:17:49.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:17:49.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:49.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:49.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.559+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338013950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:49.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.563+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:49.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.563+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:49.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.563+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:49.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.563+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:49.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.563+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800bec0 con 0x7f834c0f6280 2021-03-09T09:17:49.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.563+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338013950 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:49.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f834c0f6280 2021-03-09T09:17:49.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f834c0f6280 2021-03-09T09:17:49.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:49.567+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f834c0f6280 2021-03-09T09:17:50.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:17:49 smithi146 bash[13136]: debug 2021-03-09T09:17:49.558+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4531 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:17:54.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338013950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:17:54.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338013950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:54.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338013950 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:54.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338013950 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c04e870).stop 2021-03-09T09:17:54.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338013950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:54.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338013950 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:54.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:54.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:54.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:54.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024200 con 0x7f834c0f4690 2021-03-09T09:17:54.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011a80 con 0x7f833805c1c0 2021-03-09T09:17:54.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ede0 con 0x7f8338062ca0 2021-03-09T09:17:54.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:17:54.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:54.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:17:54.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:54.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.559+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:54.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.563+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:54.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.563+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:54.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.563+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:54.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.563+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:54.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.563+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338066e10 con 0x7f833805c1c0 2021-03-09T09:17:54.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.563+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 rx=0x7f834400c280 tx=0x7f834404ac70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:54.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f833805c1c0 2021-03-09T09:17:54.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402b760 con 0x7f833805c1c0 2021-03-09T09:17:54.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:54.571+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833805c1c0 2021-03-09T09:17:55.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:17:54 smithi146 bash[13136]: debug 2021-03-09T09:17:54.558+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4557 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:17:59.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:17:59.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:59.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 rx=0x7f834400c280 tx=0x7f834404ac70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:59.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 rx=0x7f834400c280 tx=0x7f834404ac70).stop 2021-03-09T09:17:59.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:59.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:59.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:59.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:59.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:59.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cc70 con 0x7f834c0f4690 2021-03-09T09:17:59.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f070 con 0x7f834c0f6280 2021-03-09T09:17:59.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005200 con 0x7f8338062ca0 2021-03-09T09:17:59.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:59.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:17:59.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:17:59.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:59.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:59.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:59.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:59.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806ca70 con 0x7f834c0f6280 2021-03-09T09:17:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.563+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380140f0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 rx=0x7f833404bb00 tx=0x7f8334021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.567+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032030 con 0x7f834c0f6280 2021-03-09T09:17:59.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.567+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334052660 con 0x7f834c0f6280 2021-03-09T09:17:59.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:17:59.571+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334064d20 con 0x7f834c0f6280 2021-03-09T09:18:00.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:17:59 smithi146 bash[13136]: debug 2021-03-09T09:17:59.562+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4581 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:18:04.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380140f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:18:04.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380140f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:04.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380140f0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 rx=0x7f833404bb00 tx=0x7f8334021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:04.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380140f0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 rx=0x7f833404bb00 tx=0x7f8334021000).stop 2021-03-09T09:18:04.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f83380140f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:04.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f83380140f0 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:04.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:04.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:04.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338023790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:04.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069fa0 con 0x7f834c0f4690 2021-03-09T09:18:04.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338070d20 con 0x7f833805c1c0 2021-03-09T09:18:04.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006930 con 0x7f8338062ca0 2021-03-09T09:18:04.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:18:04.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:18:04.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:04.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:04.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.563+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338023790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:04.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.567+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:04.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.567+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:04.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.567+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.567+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.567+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800bec0 con 0x7f833805c1c0 2021-03-09T09:18:04.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.567+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338023790 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 rx=0x7f833c060df0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:04.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.567+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c068070 con 0x7f833805c1c0 2021-03-09T09:18:04.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.567+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c025ea0 con 0x7f833805c1c0 2021-03-09T09:18:04.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:04.571+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f833805c1c0 2021-03-09T09:18:05.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:18:04 smithi146 bash[13136]: debug 2021-03-09T09:18:04.562+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4608 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:18:09.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.563+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338023790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:18:09.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.563+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338023790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:09.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.563+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338023790 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 rx=0x7f833c060df0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:09.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.563+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338023790 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 rx=0x7f833c060df0 tx=0x7f833c04e870).stop 2021-03-09T09:18:09.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338023790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:09.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.563+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338023790 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:09.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.563+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:09.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.563+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:09.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.563+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:09.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.563+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:18:09.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.563+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:09.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a120 con 0x7f834c0f4690 2021-03-09T09:18:09.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ef90 con 0x7f834c0f6280 2021-03-09T09:18:09.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.563+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069fa0 con 0x7f8338062ca0 2021-03-09T09:18:09.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.567+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:18:09.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.567+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:09.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.567+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:09.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.571+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:09.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.571+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:09.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.571+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:09.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.571+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:09.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.571+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338066e10 con 0x7f834c0f6280 2021-03-09T09:18:09.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.571+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800cbb0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 rx=0x7f833c056f80 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:09.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.571+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f834c0f6280 2021-03-09T09:18:09.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.571+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f030 con 0x7f834c0f6280 2021-03-09T09:18:09.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:09.575+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05b410 con 0x7f834c0f6280 2021-03-09T09:18:10.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:18:09 smithi146 bash[13136]: debug 2021-03-09T09:18:09.562+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4627 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:18:14.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:18:14.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:14.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800cbb0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 rx=0x7f833c056f80 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:14.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800cbb0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 rx=0x7f833c056f80 tx=0x7f833c04e870).stop 2021-03-09T09:18:14.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800cbb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:14.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800cbb0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:14.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:14.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:14.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:14.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f834c0f4690 2021-03-09T09:18:14.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a790 con 0x7f833805c1c0 2021-03-09T09:18:14.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ef90 con 0x7f8338062ca0 2021-03-09T09:18:14.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:18:14.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:18:14.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:14.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:14.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:14.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:14.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:14.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:14.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:14.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380051b0 con 0x7f833805c1c0 2021-03-09T09:18:14.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 rx=0x7f833c0195b0 tx=0x7f833c024040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:14.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f833805c1c0 2021-03-09T09:18:14.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f833805c1c0 2021-03-09T09:18:14.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:14.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f833805c1c0 2021-03-09T09:18:15.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:18:14 smithi146 bash[13136]: debug 2021-03-09T09:18:14.566+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4656 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:18:19.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:18:19.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:19.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 rx=0x7f833c0195b0 tx=0x7f833c024040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:19.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 rx=0x7f833c0195b0 tx=0x7f833c024040).stop 2021-03-09T09:18:19.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:19.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:19.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:19.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:19.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:19.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e120 con 0x7f834c0f4690 2021-03-09T09:18:19.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380084a0 con 0x7f834c0f6280 2021-03-09T09:18:19.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066610 con 0x7f8338062ca0 2021-03-09T09:18:19.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:18:19.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:19.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:18:19.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:19.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:19.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:19.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:19.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:19.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:19.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380068e0 con 0x7f834c0f6280 2021-03-09T09:18:19.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009ae0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 rx=0x7f833c054040 tx=0x7f833c024040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:19.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c069e70 con 0x7f834c0f6280 2021-03-09T09:18:19.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f030 con 0x7f834c0f6280 2021-03-09T09:18:19.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:19.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f834c0f6280 2021-03-09T09:18:20.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:18:19 smithi146 bash[13136]: debug 2021-03-09T09:18:19.566+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4681 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:18:24.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:18:24.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:24.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009ae0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 rx=0x7f833c054040 tx=0x7f833c024040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:24.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009ae0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 rx=0x7f833c054040 tx=0x7f833c024040).stop 2021-03-09T09:18:24.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:24.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009ae0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:24.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:24.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:24.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:24.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005e40 con 0x7f834c0f4690 2021-03-09T09:18:24.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061ca0 con 0x7f833805c1c0 2021-03-09T09:18:24.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f070 con 0x7f8338062ca0 2021-03-09T09:18:24.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.566+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:18:24.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.566+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:24.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.570+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:18:24.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.570+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:24.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:24.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:24.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:24.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:24.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:24.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065c40 con 0x7f833805c1c0 2021-03-09T09:18:24.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 rx=0x7f833c063050 tx=0x7f833c024040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:24.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c069e30 con 0x7f833805c1c0 2021-03-09T09:18:24.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c062350 con 0x7f833805c1c0 2021-03-09T09:18:24.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:24.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f833805c1c0 2021-03-09T09:18:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:18:24 smithi146 bash[13136]: debug 2021-03-09T09:18:24.570+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4707 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:18:29.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:18:29.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:29.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 rx=0x7f833c063050 tx=0x7f833c024040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:29.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 rx=0x7f833c063050 tx=0x7f833c024040).stop 2021-03-09T09:18:29.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:29.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:29.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:29.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:29.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:29.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380268d0 con 0x7f834c0f4690 2021-03-09T09:18:29.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025a70 con 0x7f834c0f6280 2021-03-09T09:18:29.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f8338062ca0 2021-03-09T09:18:29.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:18:29.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:29.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:29.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:18:29.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:29.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:29.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:29.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:29.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:29.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800bec0 con 0x7f834c0f6280 2021-03-09T09:18:29.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.570+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009a20 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:29.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f834c0f6280 2021-03-09T09:18:29.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404bd80 con 0x7f834c0f6280 2021-03-09T09:18:29.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:29.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402da60 con 0x7f834c0f6280 2021-03-09T09:18:30.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:18:29 smithi146 bash[13136]: debug 2021-03-09T09:18:29.570+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4731 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:18:34.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:18:34.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:34.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009a20 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:34.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009a20 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334021000).stop 2021-03-09T09:18:34.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:34.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009a20 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:34.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:34.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:34.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338004780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:34.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a750 con 0x7f834c0f4690 2021-03-09T09:18:34.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e0e0 con 0x7f833805c1c0 2021-03-09T09:18:34.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380084a0 con 0x7f8338062ca0 2021-03-09T09:18:34.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:18:34.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:34.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:18:34.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:34.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:34.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:34.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:34.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:34.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:34.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800bec0 con 0x7f833805c1c0 2021-03-09T09:18:34.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338004780 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c024040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:34.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f833805c1c0 2021-03-09T09:18:34.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00d280 con 0x7f833805c1c0 2021-03-09T09:18:34.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:34.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0068e0 con 0x7f833805c1c0 2021-03-09T09:18:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:18:34 smithi146 bash[13136]: debug 2021-03-09T09:18:34.569+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4757 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:18:39.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:18:39.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:39.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338004780 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c024040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:39.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338004780 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c024040).stop 2021-03-09T09:18:39.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338004780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:39.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338004780 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:39.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:39.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:39.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:39.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d740 con 0x7f834c0f4690 2021-03-09T09:18:39.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028410 con 0x7f834c0f6280 2021-03-09T09:18:39.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380289d0 con 0x7f8338062ca0 2021-03-09T09:18:39.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.570+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:18:39.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.570+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:39.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.570+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:18:39.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.570+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:39.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:39.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:39.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:39.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:39.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:39.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801aa90 con 0x7f834c0f6280 2021-03-09T09:18:39.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009a20 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c024040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:39.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0068e0 con 0x7f834c0f6280 2021-03-09T09:18:39.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c006a40 con 0x7f834c0f6280 2021-03-09T09:18:39.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:39.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04bcb0 con 0x7f834c0f6280 2021-03-09T09:18:40.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:18:39 smithi146 bash[13136]: debug 2021-03-09T09:18:39.573+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4778 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:18:44.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:18:44.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:44.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009a20 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c024040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:44.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009a20 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c024040).stop 2021-03-09T09:18:44.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338009a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:44.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338009a20 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:44.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:44.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:44.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:44.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380055a0 con 0x7f834c0f4690 2021-03-09T09:18:44.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb90 con 0x7f833805c1c0 2021-03-09T09:18:44.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bd90 con 0x7f8338062ca0 2021-03-09T09:18:44.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:18:44.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:18:44.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:44.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:44.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.574+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:44.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.578+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:44.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.578+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:44.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.578+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:44.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.578+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:44.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.578+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800bec0 con 0x7f833805c1c0 2021-03-09T09:18:44.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.578+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801d000 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f8334021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:44.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f833805c1c0 2021-03-09T09:18:44.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340524a0 con 0x7f833805c1c0 2021-03-09T09:18:44.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:44.582+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402dce0 con 0x7f833805c1c0 2021-03-09T09:18:45.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:18:44 smithi146 bash[13136]: debug 2021-03-09T09:18:44.573+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4807 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:18:49.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:18:49.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:49.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801d000 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f8334021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:49.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801d000 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f8334021000).stop 2021-03-09T09:18:49.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801d000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:49.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801d000 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:49.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:49.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:49.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:49.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380084a0 con 0x7f834c0f4690 2021-03-09T09:18:49.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f834c0f6280 2021-03-09T09:18:49.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068ff0 con 0x7f8338062ca0 2021-03-09T09:18:49.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:18:49.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:49.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:18:49.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:49.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:49.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.578+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:49.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.578+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:49.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.578+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:49.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.578+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:49.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.578+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024f20 con 0x7f834c0f6280 2021-03-09T09:18:49.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.578+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021080 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 rx=0x7f833c00d4b0 tx=0x7f833c024040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:49.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0572b0 con 0x7f834c0f6280 2021-03-09T09:18:49.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f834c0f6280 2021-03-09T09:18:49.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:49.582+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f834c0f6280 2021-03-09T09:18:50.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:18:49 smithi146 bash[13136]: debug 2021-03-09T09:18:49.573+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4831 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:18:54.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:18:54.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:54.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021080 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 rx=0x7f833c00d4b0 tx=0x7f833c024040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:54.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021080 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 rx=0x7f833c00d4b0 tx=0x7f833c024040).stop 2021-03-09T09:18:54.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:54.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021080 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:54.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:54.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338021570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:54.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:54.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061ca0 con 0x7f834c0f4690 2021-03-09T09:18:54.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067600 con 0x7f833805c1c0 2021-03-09T09:18:54.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380055a0 con 0x7f8338062ca0 2021-03-09T09:18:54.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:18:54.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:18:54.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:54.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:54.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338021570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:54.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:54.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:54.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:54.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:54.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380112d0 con 0x7f833805c1c0 2021-03-09T09:18:54.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338021570 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404ac70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:54.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f833805c1c0 2021-03-09T09:18:54.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f833805c1c0 2021-03-09T09:18:54.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:54.582+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404ad20 con 0x7f833805c1c0 2021-03-09T09:18:55.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:18:54 smithi146 bash[13136]: debug 2021-03-09T09:18:54.577+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4853 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:18:59.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338021570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:18:59.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338021570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:59.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338021570 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404ac70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:59.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338021570 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404ac70).stop 2021-03-09T09:18:59.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338021570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:59.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338021570 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:59.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:59.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:18:59.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:59.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:59.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:59.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380205c0 con 0x7f834c0f4690 2021-03-09T09:18:59.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d540 con 0x7f834c0f6280 2021-03-09T09:18:59.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380055a0 con 0x7f8338062ca0 2021-03-09T09:18:59.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:18:59.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:59.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:59.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:59.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:59.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:59.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.578+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:59.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.582+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338015940 con 0x7f834c0f6280 2021-03-09T09:18:59.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.582+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800e760 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834404ac70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:59.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.582+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344016580 con 0x7f834c0f6280 2021-03-09T09:18:59.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.582+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344016580 con 0x7f834c0f6280 2021-03-09T09:18:59.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:18:59.586+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404b160 con 0x7f834c0f6280 2021-03-09T09:19:00.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:18:59 smithi146 bash[13136]: debug 2021-03-09T09:18:59.577+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4881 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:19:04.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:19:04.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:04.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800e760 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834404ac70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:04.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800e760 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834404ac70).stop 2021-03-09T09:19:04.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833800e760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:04.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833800e760 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:04.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:04.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338014c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ec10 con 0x7f834c0f4690 2021-03-09T09:19:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068ff0 con 0x7f833805c1c0 2021-03-09T09:19:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d540 con 0x7f8338062ca0 2021-03-09T09:19:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:19:04.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:04.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:19:04.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:04.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.578+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338014c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.582+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.582+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.582+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.582+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:04.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.582+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024f20 con 0x7f833805c1c0 2021-03-09T09:19:04.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.582+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338014c90 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 rx=0x7f834407afd0 tx=0x7f834404ac70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:04.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.582+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400c0c0 con 0x7f833805c1c0 2021-03-09T09:19:04.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.582+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344027e90 con 0x7f833805c1c0 2021-03-09T09:19:04.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:04.586+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402b430 con 0x7f833805c1c0 2021-03-09T09:19:05.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:19:04 smithi146 bash[13136]: debug 2021-03-09T09:19:04.577+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4906 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:19:09.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338014c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:19:09.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338014c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:09.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338014c90 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 rx=0x7f834407afd0 tx=0x7f834404ac70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:09.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338014c90 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 rx=0x7f834407afd0 tx=0x7f834404ac70).stop 2021-03-09T09:19:09.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338014c90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:09.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338014c90 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:09.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:09.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:09.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:09.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:19:09.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:09.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:09.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:19:09.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:09.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ae10 con 0x7f834c0f4690 2021-03-09T09:19:09.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c200 con 0x7f834c0f6280 2021-03-09T09:19:09.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024250 con 0x7f8338062ca0 2021-03-09T09:19:09.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:09.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:09.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:09.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:09.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380112d0 con 0x7f834c0f6280 2021-03-09T09:19:09.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.581+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021790 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f8334021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:09.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f834c0f6280 2021-03-09T09:19:09.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340526b0 con 0x7f834c0f6280 2021-03-09T09:19:09.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:09.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334007850 con 0x7f834c0f6280 2021-03-09T09:19:10.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:19:09 smithi146 bash[13136]: debug 2021-03-09T09:19:09.577+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4927 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:19:14.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338021790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:19:14.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338021790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:14.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021790 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f8334021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:14.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021790 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f8334021000).stop 2021-03-09T09:19:14.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338021790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:14.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021790 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:14.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338063590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:14.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:14.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:14.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064e90 con 0x7f834c0f4690 2021-03-09T09:19:14.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:19:14.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d540 con 0x7f833805c1c0 2021-03-09T09:19:14.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ae10 con 0x7f8338062ca0 2021-03-09T09:19:14.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:14.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:19:14.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:14.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338063590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:14.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:14.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:14.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:14.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:14.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010a60 con 0x7f833805c1c0 2021-03-09T09:19:14.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.581+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338063590 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334053000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:14.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032040 con 0x7f833805c1c0 2021-03-09T09:19:14.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334062e70 con 0x7f833805c1c0 2021-03-09T09:19:14.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:14.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402dc70 con 0x7f833805c1c0 2021-03-09T09:19:15.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:19:14 smithi146 bash[13136]: debug 2021-03-09T09:19:14.581+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4955 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:19:19.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338063590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:19:19.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338063590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338063590 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334053000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:19.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338063590 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334053000).stop 2021-03-09T09:19:19.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338063590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:19.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338063590 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:19:19.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:19:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:19.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f834c0f4690 2021-03-09T09:19:19.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f930 con 0x7f834c0f6280 2021-03-09T09:19:19.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b2c0 con 0x7f8338062ca0 2021-03-09T09:19:19.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:19.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:19.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:19.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:19.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380180f0 con 0x7f834c0f6280 2021-03-09T09:19:19.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801d000 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 rx=0x7f834404b030 tx=0x7f834404ac70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:19.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f834c0f6280 2021-03-09T09:19:19.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402b760 con 0x7f834c0f6280 2021-03-09T09:19:19.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:19.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402b220 con 0x7f834c0f6280 2021-03-09T09:19:20.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:19:19 smithi146 bash[13136]: debug 2021-03-09T09:19:19.581+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 4981 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:19:24.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.581+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:19:24.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.581+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:24.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801d000 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 rx=0x7f834404b030 tx=0x7f834404ac70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:24.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801d000 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 rx=0x7f834404b030 tx=0x7f834404ac70).stop 2021-03-09T09:19:24.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801d000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:24.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801d000 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:24.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:24.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338067ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:24.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:24.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.585+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:19:24.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.585+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:24.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806ce20 con 0x7f834c0f4690 2021-03-09T09:19:24.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067650 con 0x7f833805c1c0 2021-03-09T09:19:24.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.585+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:19:24.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338067ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:24.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380051b0 con 0x7f8338062ca0 2021-03-09T09:19:24.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.585+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:24.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:24.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:24.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:24.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:24.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024f20 con 0x7f833805c1c0 2021-03-09T09:19:24.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338067ee0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 rx=0x7f834407ad90 tx=0x7f834404ac70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:24.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400c2a0 con 0x7f833805c1c0 2021-03-09T09:19:24.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404a5d0 con 0x7f833805c1c0 2021-03-09T09:19:24.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:24.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404b5f0 con 0x7f833805c1c0 2021-03-09T09:19:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:19:24 smithi146 bash[13136]: debug 2021-03-09T09:19:24.585+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5006 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:19:29.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:19:29.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:29.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338067ee0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 rx=0x7f834407ad90 tx=0x7f834404ac70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:29.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338067ee0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 rx=0x7f834407ad90 tx=0x7f834404ac70).stop 2021-03-09T09:19:29.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f8338067ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:29.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f8338067ee0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:29.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:29.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:29.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:29.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b2c0 con 0x7f834c0f4690 2021-03-09T09:19:29.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d540 con 0x7f834c0f6280 2021-03-09T09:19:29.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806ce20 con 0x7f8338062ca0 2021-03-09T09:19:29.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:19:29.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:19:29.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:29.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:29.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:29.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:29.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:29.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:29.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:29.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010a60 con 0x7f834c0f6280 2021-03-09T09:19:29.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.589+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801d000 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 rx=0x7f8344077d10 tx=0x7f834400c1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:29.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344077920 con 0x7f834c0f6280 2021-03-09T09:19:29.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344077740 con 0x7f834c0f6280 2021-03-09T09:19:29.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:29.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400e140 con 0x7f834c0f6280 2021-03-09T09:19:30.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:19:29 smithi146 bash[13136]: debug 2021-03-09T09:19:29.585+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5031 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:19:34.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:19:34.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:34.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801d000 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 rx=0x7f8344077d10 tx=0x7f834400c1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:34.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801d000 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 rx=0x7f8344077d10 tx=0x7f834400c1f0).stop 2021-03-09T09:19:34.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801d000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:34.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801d000 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:34.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:34.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:34.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833800b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:34.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:19:34.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ae10 con 0x7f834c0f4690 2021-03-09T09:19:34.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:34.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028410 con 0x7f833805c1c0 2021-03-09T09:19:34.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800df20 con 0x7f8338062ca0 2021-03-09T09:19:34.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833800b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:34.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:19:34.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:34.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.589+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:34.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:34.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.589+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:34.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:34.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.589+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024f20 con 0x7f833805c1c0 2021-03-09T09:19:34.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833800b620 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 rx=0x7f833c0623f0 tx=0x7f833c024040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:34.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f833805c1c0 2021-03-09T09:19:34.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833805c1c0 2021-03-09T09:19:34.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:34.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06ab90 con 0x7f833805c1c0 2021-03-09T09:19:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:19:34 smithi146 bash[13136]: debug 2021-03-09T09:19:34.584+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5055 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:19:39.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:19:39.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:39.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833800b620 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 rx=0x7f833c0623f0 tx=0x7f833c024040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:39.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833800b620 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 rx=0x7f833c0623f0 tx=0x7f833c024040).stop 2021-03-09T09:19:39.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833800b620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:39.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833800b620 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:39.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:39.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:39.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:39.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:19:39.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:39.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:19:39.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:39.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb00 con 0x7f834c0f4690 2021-03-09T09:19:39.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061ca0 con 0x7f834c0f6280 2021-03-09T09:19:39.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e7d0 con 0x7f8338062ca0 2021-03-09T09:19:39.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.589+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:39.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.597+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:39.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:39.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.597+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:39.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:39.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.597+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028410 con 0x7f834c0f6280 2021-03-09T09:19:39.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801cfb0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 rx=0x7f834401f9c0 tx=0x7f834400c1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:39.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029030 con 0x7f834c0f6280 2021-03-09T09:19:39.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440307d0 con 0x7f834c0f6280 2021-03-09T09:19:39.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:39.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f834c0f6280 2021-03-09T09:19:40.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:19:39 smithi146 bash[13136]: debug 2021-03-09T09:19:39.588+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5080 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:19:44.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:19:44.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:44.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801cfb0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 rx=0x7f834401f9c0 tx=0x7f834400c1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:44.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801cfb0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 rx=0x7f834401f9c0 tx=0x7f834400c1f0).stop 2021-03-09T09:19:44.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801cfb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:44.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801cfb0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:44.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:44.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:44.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:44.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:19:44.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:19:44.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:44.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:44.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806ce20 con 0x7f834c0f4690 2021-03-09T09:19:44.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010600 con 0x7f833805c1c0 2021-03-09T09:19:44.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a300 con 0x7f8338062ca0 2021-03-09T09:19:44.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:44.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:44.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.589+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:44.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.593+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:44.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.593+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:44.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.593+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802bb90 con 0x7f833805c1c0 2021-03-09T09:19:44.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.593+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801d4a0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f833404bfb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:44.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f833805c1c0 2021-03-09T09:19:44.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402da60 con 0x7f833805c1c0 2021-03-09T09:19:44.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:44.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334007850 con 0x7f833805c1c0 2021-03-09T09:19:45.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:19:44 smithi146 bash[13136]: debug 2021-03-09T09:19:44.588+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5105 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:19:49.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:19:49.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:49.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801d4a0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f833404bfb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:49.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801d4a0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f833404bfb0).stop 2021-03-09T09:19:49.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801d4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:49.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801d4a0 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:49.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:49.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:49.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:49.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:19:49.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:49.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:19:49.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:49.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f834c0f4690 2021-03-09T09:19:49.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013a40 con 0x7f834c0f6280 2021-03-09T09:19:49.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb00 con 0x7f8338062ca0 2021-03-09T09:19:49.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.589+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:49.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.593+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:49.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:49.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.593+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:49.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:49.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.593+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065ff0 con 0x7f834c0f6280 2021-03-09T09:19:49.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a0d0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 rx=0x7f8344003fd0 tx=0x7f834400c1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:49.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029030 con 0x7f834c0f6280 2021-03-09T09:19:49.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344018e90 con 0x7f834c0f6280 2021-03-09T09:19:49.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:49.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344077de0 con 0x7f834c0f6280 2021-03-09T09:19:50.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:19:49 smithi146 bash[13136]: debug 2021-03-09T09:19:49.592+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5131 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:19:54.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.589+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:19:54.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.593+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:54.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a0d0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 rx=0x7f8344003fd0 tx=0x7f834400c1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:54.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a0d0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 rx=0x7f8344003fd0 tx=0x7f834400c1f0).stop 2021-03-09T09:19:54.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801a0d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:54.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a0d0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:54.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:54.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:54.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:54.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.593+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:19:54.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.593+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:54.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:19:54.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:54.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e7d0 con 0x7f834c0f4690 2021-03-09T09:19:54.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb00 con 0x7f833805c1c0 2021-03-09T09:19:54.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028d60 con 0x7f8338062ca0 2021-03-09T09:19:54.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:54.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.597+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:54.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:54.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.597+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:54.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:54.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.597+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065ff0 con 0x7f833805c1c0 2021-03-09T09:19:54.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 rx=0x7f8344077ae0 tx=0x7f834400c1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:54.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029040 con 0x7f833805c1c0 2021-03-09T09:19:54.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344018e90 con 0x7f833805c1c0 2021-03-09T09:19:54.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:54.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833805c1c0 2021-03-09T09:19:55.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:19:54 smithi146 bash[13136]: debug 2021-03-09T09:19:54.592+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5155 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:19:59.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:19:59.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:59.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 rx=0x7f8344077ae0 tx=0x7f834400c1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:59.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 rx=0x7f8344077ae0 tx=0x7f834400c1f0).stop 2021-03-09T09:19:59.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801dae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:59.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:59.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:59.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:59.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:59.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:19:59.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:59.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:19:59.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:59.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010010 con 0x7f834c0f4690 2021-03-09T09:19:59.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064e90 con 0x7f834c0f6280 2021-03-09T09:19:59.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d540 con 0x7f8338062ca0 2021-03-09T09:19:59.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:59.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.597+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:59.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.597+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:59.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.597+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:59.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.597+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:59.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.597+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005750 con 0x7f834c0f6280 2021-03-09T09:19:59.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.597+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a0d0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 rx=0x7f833800e490 tx=0x7f833c012850).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:59.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f834c0f6280 2021-03-09T09:19:59.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f834c0f6280 2021-03-09T09:19:59.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:19:59.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f834c0f6280 2021-03-09T09:20:00.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:19:59 smithi146 bash[13136]: debug 2021-03-09T09:19:59.592+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5179 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:20:04.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:20:04.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:04.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a0d0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 rx=0x7f833800e490 tx=0x7f833c012850).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:04.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a0d0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 rx=0x7f833800e490 tx=0x7f833c012850).stop 2021-03-09T09:20:04.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833801a0d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:04.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833801a0d0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:04.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:04.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:04.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:04.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.593+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:20:04.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.593+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:04.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.593+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:20:04.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:04.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806f640 con 0x7f834c0f4690 2021-03-09T09:20:04.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb00 con 0x7f833805c1c0 2021-03-09T09:20:04.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028570 con 0x7f8338062ca0 2021-03-09T09:20:04.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.597+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:04.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.597+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:04.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.597+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:04.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.597+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:04.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.597+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:04.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.597+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005750 con 0x7f833805c1c0 2021-03-09T09:20:04.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.597+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c074000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:04.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0553a0 con 0x7f833805c1c0 2021-03-09T09:20:04.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04de90 con 0x7f833805c1c0 2021-03-09T09:20:04.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:04.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0049d0 con 0x7f833805c1c0 2021-03-09T09:20:05.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:20:04 smithi146 bash[13136]: debug 2021-03-09T09:20:04.596+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5206 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:20:09.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.592+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:20:09.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.596+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:09.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.596+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c074000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:09.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.596+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c074000).stop 2021-03-09T09:20:09.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.596+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801dae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:09.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.596+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:09.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.596+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:09.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.596+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:20:09.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.596+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:09.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.596+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:09.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.596+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:09.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.596+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a300 con 0x7f834c0f4690 2021-03-09T09:20:09.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.596+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:20:09.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.596+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010010 con 0x7f834c0f6280 2021-03-09T09:20:09.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.596+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8338062ca0 2021-03-09T09:20:09.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.596+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:09.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:09.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.600+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:09.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:09.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.600+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:09.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:09.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.600+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338016240 con 0x7f834c0f6280 2021-03-09T09:20:09.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c074000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:09.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01a6e0 con 0x7f834c0f6280 2021-03-09T09:20:09.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01ad20 con 0x7f834c0f6280 2021-03-09T09:20:09.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:09.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c020a60 con 0x7f834c0f6280 2021-03-09T09:20:10.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:20:09 smithi146 bash[13136]: debug 2021-03-09T09:20:09.596+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5227 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:20:14.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:20:14.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:14.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c074000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:14.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c074000).stop 2021-03-09T09:20:14.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:14.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833807dc80 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:14.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:14.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:20:14.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:14.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:14.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:14.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:20:14.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:14.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833805eb50 con 0x7f834c0f4690 2021-03-09T09:20:14.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380233e0 con 0x7f833805c1c0 2021-03-09T09:20:14.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e450 con 0x7f8338062ca0 2021-03-09T09:20:14.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:14.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:14.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:14.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:14.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:14.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338025a20 con 0x7f833805c1c0 2021-03-09T09:20:14.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7f833c0419c0 tx=0x7f833c074000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:14.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c035610 con 0x7f833805c1c0 2021-03-09T09:20:14.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f833805c1c0 2021-03-09T09:20:14.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:14.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c058970 con 0x7f833805c1c0 2021-03-09T09:20:15.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:20:14 smithi146 bash[13136]: debug 2021-03-09T09:20:14.600+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5256 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:20:19.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.596+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:20:19.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.596+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:19.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.596+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7f833c0419c0 tx=0x7f833c074000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:19.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.596+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7f833c0419c0 tx=0x7f833c074000).stop 2021-03-09T09:20:19.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.596+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801dae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.596+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.596+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.596+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.596+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:19.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.596+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016440 con 0x7f834c0f4690 2021-03-09T09:20:19.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.596+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380221b0 con 0x7f834c0f6280 2021-03-09T09:20:19.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.596+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b2c0 con 0x7f8338062ca0 2021-03-09T09:20:19.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.596+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:20:19.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.596+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:19.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.596+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:19.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.600+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:20:19.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.600+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:19.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.600+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:19.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:19.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.600+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:19.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:19.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.600+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338061530 con 0x7f834c0f6280 2021-03-09T09:20:19.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccb0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 rx=0x7f83380129d0 tx=0x7f833c074000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:19.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c035610 con 0x7f834c0f6280 2021-03-09T09:20:19.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f834c0f6280 2021-03-09T09:20:19.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:19.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04de90 con 0x7f834c0f6280 2021-03-09T09:20:20.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:20:19 smithi146 bash[13136]: debug 2021-03-09T09:20:19.600+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5280 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:20:24.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:20:24.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:24.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccb0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 rx=0x7f83380129d0 tx=0x7f833c074000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:24.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccb0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 rx=0x7f83380129d0 tx=0x7f833c074000).stop 2021-03-09T09:20:24.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802ccb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:24.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccb0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:24.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:24.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:24.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:24.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380233e0 con 0x7f834c0f4690 2021-03-09T09:20:24.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e450 con 0x7f833805c1c0 2021-03-09T09:20:24.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016440 con 0x7f8338062ca0 2021-03-09T09:20:24.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:20:24.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:24.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:20:24.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:24.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:24.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.608+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:24.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.608+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:24.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.608+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:24.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.608+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:24.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.608+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380068e0 con 0x7f833805c1c0 2021-03-09T09:20:24.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.608+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 rx=0x7f833c01d040 tx=0x7f833c074000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:24.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05f3e0 con 0x7f833805c1c0 2021-03-09T09:20:24.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f833805c1c0 2021-03-09T09:20:24.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:24.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c011870 con 0x7f833805c1c0 2021-03-09T09:20:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:20:24 smithi146 bash[13136]: debug 2021-03-09T09:20:24.600+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5306 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:20:29.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:20:29.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:29.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 rx=0x7f833c01d040 tx=0x7f833c074000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:29.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 rx=0x7f833c01d040 tx=0x7f833c074000).stop 2021-03-09T09:20:29.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801dae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:29.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dae0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:29.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:29.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:29.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:29.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064e90 con 0x7f834c0f4690 2021-03-09T09:20:29.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cf70 con 0x7f834c0f6280 2021-03-09T09:20:29.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380233e0 con 0x7f8338062ca0 2021-03-09T09:20:29.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:20:29.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:29.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:20:29.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:29.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:29.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:29.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:29.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:29.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:29.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062890 con 0x7f834c0f6280 2021-03-09T09:20:29.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.604+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccb0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833404bfb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:29.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032030 con 0x7f834c0f6280 2021-03-09T09:20:29.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402df00 con 0x7f834c0f6280 2021-03-09T09:20:29.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:29.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d220 con 0x7f834c0f6280 2021-03-09T09:20:30.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:20:29 smithi146 bash[13136]: debug 2021-03-09T09:20:29.604+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5329 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:20:34.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:20:34.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:34.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccb0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833404bfb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:34.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccb0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833404bfb0).stop 2021-03-09T09:20:34.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802ccb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:34.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ccb0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:34.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:34.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:34.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:34.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f834c0f4690 2021-03-09T09:20:34.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023cf0 con 0x7f833805c1c0 2021-03-09T09:20:34.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e3b0 con 0x7f8338062ca0 2021-03-09T09:20:34.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:20:34.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:34.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:20:34.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:34.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:34.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:34.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:34.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:34.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:34.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.604+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024f20 con 0x7f833805c1c0 2021-03-09T09:20:34.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.608+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dac0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 rx=0x7f833c035520 tx=0x7f833c074000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:34.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f833805c1c0 2021-03-09T09:20:34.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c069630 con 0x7f833805c1c0 2021-03-09T09:20:34.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:34.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00d280 con 0x7f833805c1c0 2021-03-09T09:20:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:20:34 smithi146 bash[13136]: debug 2021-03-09T09:20:34.604+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5355 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:20:39.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:20:39.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:39.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dac0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 rx=0x7f833c035520 tx=0x7f833c074000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:39.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dac0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 rx=0x7f833c035520 tx=0x7f833c074000).stop 2021-03-09T09:20:39.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 msgr2=0x7f833801dac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:39.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805c1c0 0x7f833801dac0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:39.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:39.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:39.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338063ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:39.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e410 con 0x7f834c0f4690 2021-03-09T09:20:39.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e450 con 0x7f834c0f6280 2021-03-09T09:20:39.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380233e0 con 0x7f8338062ca0 2021-03-09T09:20:39.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:20:39.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:39.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:20:39.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:39.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338063ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:39.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:39.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:39.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:39.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:39.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.604+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062890 con 0x7f834c0f6280 2021-03-09T09:20:39.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.608+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338063ef0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 rx=0x7f833c02c630 tx=0x7f833c074000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:39.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f834c0f6280 2021-03-09T09:20:39.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f834c0f6280 2021-03-09T09:20:39.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:39.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f834c0f6280 2021-03-09T09:20:40.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:20:39 smithi146 bash[13136]: debug 2021-03-09T09:20:39.607+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5379 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:20:44.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338063ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:20:44.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338063ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:44.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338063ef0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 rx=0x7f833c02c630 tx=0x7f833c074000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:44.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338063ef0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 rx=0x7f833c02c630 tx=0x7f833c074000).stop 2021-03-09T09:20:44.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338063ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:44.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338063ef0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:44.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:44.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:44.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:44.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022e10 con 0x7f834c0f4690 2021-03-09T09:20:44.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023cf0 con 0x7f833801bab0 2021-03-09T09:20:44.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067650 con 0x7f8338062ca0 2021-03-09T09:20:44.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:20:44.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:44.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:20:44.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:44.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:44.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:44.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:44.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:44.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:44.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.608+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028520 con 0x7f833801bab0 2021-03-09T09:20:44.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.612+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dee0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 rx=0x7f834402b640 tx=0x7f834400c1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:44.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f833801bab0 2021-03-09T09:20:44.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833801bab0 2021-03-09T09:20:44.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:44.616+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400d450 con 0x7f833801bab0 2021-03-09T09:20:45.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:20:44 smithi146 bash[13136]: debug 2021-03-09T09:20:44.607+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5406 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:20:49.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:20:49.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:49.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dee0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 rx=0x7f834402b640 tx=0x7f834400c1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:49.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dee0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 rx=0x7f834402b640 tx=0x7f834400c1f0).stop 2021-03-09T09:20:49.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:49.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dee0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:49.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:49.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:49.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:49.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380233e0 con 0x7f834c0f4690 2021-03-09T09:20:49.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014850 con 0x7f834c0f6280 2021-03-09T09:20:49.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004b60 con 0x7f8338062ca0 2021-03-09T09:20:49.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:20:49.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:49.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:20:49.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:49.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:49.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:49.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:49.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:49.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:49.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338069370 con 0x7f834c0f6280 2021-03-09T09:20:49.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.608+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021750 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834400c1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:49.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344072190 con 0x7f834c0f6280 2021-03-09T09:20:49.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f834c0f6280 2021-03-09T09:20:49.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:49.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402b220 con 0x7f834c0f6280 2021-03-09T09:20:50.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:20:49 smithi146 bash[13136]: debug 2021-03-09T09:20:49.607+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5429 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:20:54.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338021750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:20:54.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338021750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:54.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021750 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834400c1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:54.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021750 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834400c1f0).stop 2021-03-09T09:20:54.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338021750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:54.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338021750 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:54.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:54.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:54.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:54.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e120 con 0x7f834c0f4690 2021-03-09T09:20:54.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e450 con 0x7f833801bab0 2021-03-09T09:20:54.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380233e0 con 0x7f8338062ca0 2021-03-09T09:20:54.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:20:54.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:54.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:54.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:20:54.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:54.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:54.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:54.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:54.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:54.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338013200 con 0x7f833801bab0 2021-03-09T09:20:54.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833404bfb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:54.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f833801bab0 2021-03-09T09:20:54.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.611+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007c30 con 0x7f833801bab0 2021-03-09T09:20:54.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:54.615+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d5d0 con 0x7f833801bab0 2021-03-09T09:20:55.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:20:54 smithi146 bash[13136]: debug 2021-03-09T09:20:54.611+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5452 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:20:59.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:20:59.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:59.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833404bfb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:59.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833404bfb0).stop 2021-03-09T09:20:59.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:59.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:59.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:59.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:59.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:59.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:20:59.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:20:59.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:59.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:59.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027990 con 0x7f834c0f4690 2021-03-09T09:20:59.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833805eb50 con 0x7f834c0f6280 2021-03-09T09:20:59.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d5c0 con 0x7f8338062ca0 2021-03-09T09:20:59.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.611+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:59.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.615+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:59.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.615+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:59.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.615+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:59.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.615+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:59.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.615+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800b2c0 con 0x7f834c0f6280 2021-03-09T09:20:59.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.615+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ca30 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 rx=0x7f8334023d00 tx=0x7f8334004f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:59.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.615+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f834c0f6280 2021-03-09T09:20:59.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.615+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334063180 con 0x7f834c0f6280 2021-03-09T09:20:59.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:20:59.619+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403acf0 con 0x7f834c0f6280 2021-03-09T09:21:00.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:20:59 smithi146 bash[13136]: debug 2021-03-09T09:20:59.611+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5478 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:21:04.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:21:04.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ca30 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 rx=0x7f8334023d00 tx=0x7f8334004f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ca30 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 rx=0x7f8334023d00 tx=0x7f8334004f50).stop 2021-03-09T09:21:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f833802ca30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f833802ca30 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:04.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:04.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:04.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:04.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014850 con 0x7f834c0f4690 2021-03-09T09:21:04.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380233e0 con 0x7f833801bab0 2021-03-09T09:21:04.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024260 con 0x7f8338062ca0 2021-03-09T09:21:04.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:21:04.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:04.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:21:04.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:04.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:04.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.615+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338008bd0 con 0x7f833801bab0 2021-03-09T09:21:04.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.619+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 rx=0x7f8338028950 tx=0x7f834400c1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:04.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.619+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f833801bab0 2021-03-09T09:21:04.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.619+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402b760 con 0x7f833801bab0 2021-03-09T09:21:04.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:04.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833801bab0 2021-03-09T09:21:05.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:21:04 smithi146 bash[13136]: debug 2021-03-09T09:21:04.615+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5506 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:21:09.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:21:09.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:09.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 rx=0x7f8338028950 tx=0x7f834400c1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:09.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 rx=0x7f8338028950 tx=0x7f834400c1f0).stop 2021-03-09T09:21:09.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:09.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:09.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:09.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:09.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338026c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:09.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023ef0 con 0x7f834c0f4690 2021-03-09T09:21:09.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018e70 con 0x7f834c0f6280 2021-03-09T09:21:09.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026760 con 0x7f8338062ca0 2021-03-09T09:21:09.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:21:09.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:09.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:21:09.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:09.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.615+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338026c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:09.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.619+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:09.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.619+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:09.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.619+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:09.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.619+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:09.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.619+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338013200 con 0x7f834c0f6280 2021-03-09T09:21:09.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.619+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338026c80 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 rx=0x7f833c053610 tx=0x7f833c074000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:09.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.619+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f834c0f6280 2021-03-09T09:21:09.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.619+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f834c0f6280 2021-03-09T09:21:09.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:09.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04df00 con 0x7f834c0f6280 2021-03-09T09:21:10.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:21:09 smithi146 bash[13136]: debug 2021-03-09T09:21:09.615+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5529 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:21:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.615+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338026c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:21:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.615+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338026c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.615+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338026c80 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 rx=0x7f833c053610 tx=0x7f833c074000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:14.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.615+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338026c80 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 rx=0x7f833c053610 tx=0x7f833c074000).stop 2021-03-09T09:21:14.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.615+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 msgr2=0x7f8338026c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:14.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.615+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f834c0f6280 0x7f8338026c80 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:14.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.615+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:14.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.615+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:14.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.615+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:14.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.615+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063970 con 0x7f834c0f4690 2021-03-09T09:21:14.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.615+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004b60 con 0x7f833801bab0 2021-03-09T09:21:14.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.615+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e450 con 0x7f8338062ca0 2021-03-09T09:21:14.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.615+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:21:14.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.615+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:14.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.619+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:21:14.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.619+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.619+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.619+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.619+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.619+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.619+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.619+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380068e0 con 0x7f833801bab0 2021-03-09T09:21:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.619+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 rx=0x7f833c0259e0 tx=0x7f833c074000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c009200 con 0x7f833801bab0 2021-03-09T09:21:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f833801bab0 2021-03-09T09:21:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:14.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c053310 con 0x7f833801bab0 2021-03-09T09:21:15.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:21:14 smithi146 bash[13136]: debug 2021-03-09T09:21:14.615+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5556 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:21:19.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:21:19.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:19.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 rx=0x7f833c0259e0 tx=0x7f833c074000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:19.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 rx=0x7f833c0259e0 tx=0x7f833c074000).stop 2021-03-09T09:21:19.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:19.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:19.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:19.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:19.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338027100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:19.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023ef0 con 0x7f834c0f4690 2021-03-09T09:21:19.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380103b0 con 0x7f8338026c80 2021-03-09T09:21:19.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021cd0 con 0x7f8338062ca0 2021-03-09T09:21:19.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:21:19.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:19.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:21:19.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:19.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.619+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338027100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:19.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.623+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:19.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.623+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:19.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.623+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:19.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.623+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:19.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.623+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024f20 con 0x7f8338026c80 2021-03-09T09:21:19.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.623+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338027100 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 rx=0x7f833c069410 tx=0x7f833c01d000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:19.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f8338026c80 2021-03-09T09:21:19.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f8338026c80 2021-03-09T09:21:19.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:19.627+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c053270 con 0x7f8338026c80 2021-03-09T09:21:20.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:21:19 smithi146 bash[13136]: debug 2021-03-09T09:21:19.619+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5579 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:21:24.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338027100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:21:24.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338027100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:24.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338027100 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 rx=0x7f833c069410 tx=0x7f833c01d000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:24.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338027100 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 rx=0x7f833c069410 tx=0x7f833c01d000).stop 2021-03-09T09:21:24.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338027100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:24.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338027100 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:24.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:24.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d540 con 0x7f834c0f4690 2021-03-09T09:21:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380103b0 con 0x7f833801bab0 2021-03-09T09:21:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011530 con 0x7f8338062ca0 2021-03-09T09:21:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:21:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:21:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.619+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:24.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.623+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:24.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.623+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:24.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.623+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:24.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.623+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:24.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.623+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801dea0 con 0x7f833801bab0 2021-03-09T09:21:24.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.623+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801a5d0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 rx=0x7f8334024b90 tx=0x7f8334004f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:24.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f833801bab0 2021-03-09T09:21:24.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334063370 con 0x7f833801bab0 2021-03-09T09:21:24.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:24.627+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d6a0 con 0x7f833801bab0 2021-03-09T09:21:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:21:24 smithi146 bash[13136]: debug 2021-03-09T09:21:24.619+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5606 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:21:29.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.619+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:21:29.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.619+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:29.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.619+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801a5d0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 rx=0x7f8334024b90 tx=0x7f8334004f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:29.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.619+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801a5d0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 rx=0x7f8334024b90 tx=0x7f8334004f50).stop 2021-03-09T09:21:29.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801a5d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:29.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801a5d0 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:29.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:29.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:29.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:29.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dfc0 con 0x7f834c0f4690 2021-03-09T09:21:29.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028080 con 0x7f8338026c80 2021-03-09T09:21:29.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027990 con 0x7f8338062ca0 2021-03-09T09:21:29.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:21:29.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:29.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:21:29.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:29.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:29.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:29.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:29.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:29.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:29.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806fd70 con 0x7f8338026c80 2021-03-09T09:21:29.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.623+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806f880 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c01d000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:29.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.627+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f8338026c80 2021-03-09T09:21:29.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.627+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8338026c80 2021-03-09T09:21:29.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:29.627+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0068e0 con 0x7f8338026c80 2021-03-09T09:21:30.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:21:29 smithi146 bash[13136]: debug 2021-03-09T09:21:29.618+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5629 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:21:34.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833806f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:21:34.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833806f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:34.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806f880 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c01d000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:34.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806f880 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c01d000).stop 2021-03-09T09:21:34.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833806f880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:34.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806f880 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:34.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:34.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:34.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:34.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f834c0f4690 2021-03-09T09:21:34.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380103b0 con 0x7f833801bab0 2021-03-09T09:21:34.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8338062ca0 2021-03-09T09:21:34.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:21:34.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:34.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:21:34.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:34.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.623+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:34.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.627+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:34.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.627+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:34.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.627+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:34.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.627+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:34.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.627+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801a9b0 con 0x7f833801bab0 2021-03-09T09:21:34.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.627+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 rx=0x7f833400f040 tx=0x7f8334004f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:34.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.627+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032040 con 0x7f833801bab0 2021-03-09T09:21:34.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.627+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404b1a0 con 0x7f833801bab0 2021-03-09T09:21:34.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:34.631+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334052a50 con 0x7f833801bab0 2021-03-09T09:21:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:21:34 smithi146 bash[13136]: debug 2021-03-09T09:21:34.622+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5656 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:21:39.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:21:39.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:39.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 rx=0x7f833400f040 tx=0x7f8334004f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:39.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 rx=0x7f833400f040 tx=0x7f8334004f50).stop 2021-03-09T09:21:39.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:39.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:39.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:39.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:39.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:39.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dfc0 con 0x7f834c0f4690 2021-03-09T09:21:39.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066610 con 0x7f8338026c80 2021-03-09T09:21:39.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f8338062ca0 2021-03-09T09:21:39.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:21:39.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:39.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:21:39.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:39.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.622+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:39.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.626+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:39.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.626+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:39.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.626+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:39.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.626+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:39.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.626+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338019ee0 con 0x7f8338026c80 2021-03-09T09:21:39.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.626+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806f880 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 rx=0x7f8344054350 tx=0x7f834400c1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:39.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834407ace0 con 0x7f8338026c80 2021-03-09T09:21:39.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f8338026c80 2021-03-09T09:21:39.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:39.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400d450 con 0x7f8338026c80 2021-03-09T09:21:40.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:21:39 smithi146 bash[13136]: debug 2021-03-09T09:21:39.622+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5679 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:21:44.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833806f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:21:44.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833806f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:44.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806f880 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 rx=0x7f8344054350 tx=0x7f834400c1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:44.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806f880 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 rx=0x7f8344054350 tx=0x7f834400c1f0).stop 2021-03-09T09:21:44.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833806f880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:44.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806f880 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:44.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:44.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:44.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:44.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f834c0f4690 2021-03-09T09:21:44.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380129b0 con 0x7f833801bab0 2021-03-09T09:21:44.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014ee0 con 0x7f8338062ca0 2021-03-09T09:21:44.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:21:44.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:44.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:21:44.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:44.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.626+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:44.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.630+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:44.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.630+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:44.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.630+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:44.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.630+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:44.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.630+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012af0 con 0x7f833801bab0 2021-03-09T09:21:44.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.630+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f8334004f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:44.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f833801bab0 2021-03-09T09:21:44.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402da60 con 0x7f833801bab0 2021-03-09T09:21:44.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:44.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833400b980 con 0x7f833801bab0 2021-03-09T09:21:45.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:21:44 smithi146 bash[13136]: debug 2021-03-09T09:21:44.626+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5705 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:21:49.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:21:49.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:49.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f8334004f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:49.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f8334004f50).stop 2021-03-09T09:21:49.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:49.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:49.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:49.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:49.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:49.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f834c0f4690 2021-03-09T09:21:49.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005990 con 0x7f8338026c80 2021-03-09T09:21:49.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dfc0 con 0x7f8338062ca0 2021-03-09T09:21:49.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:21:49.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:49.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:21:49.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:49.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.626+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:49.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.630+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:49.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.630+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:49.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.630+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:49.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.630+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:49.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.630+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010060 con 0x7f8338026c80 2021-03-09T09:21:49.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.630+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338025e80 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 rx=0x7f83440032f0 tx=0x7f834400c1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:49.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029030 con 0x7f8338026c80 2021-03-09T09:21:49.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338026c80 2021-03-09T09:21:49.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:49.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400d450 con 0x7f8338026c80 2021-03-09T09:21:50.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:21:49 smithi146 bash[13136]: debug 2021-03-09T09:21:49.626+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5729 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:21:54.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.626+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:21:54.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.626+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:54.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.626+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338025e80 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 rx=0x7f83440032f0 tx=0x7f834400c1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:54.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.626+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338025e80 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 rx=0x7f83440032f0 tx=0x7f834400c1f0).stop 2021-03-09T09:21:54.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:54.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.630+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338025e80 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:54.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.630+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:54.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.630+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:54.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.630+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:54.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009db0 con 0x7f834c0f4690 2021-03-09T09:21:54.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f833801bab0 2021-03-09T09:21:54.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063970 con 0x7f8338062ca0 2021-03-09T09:21:54.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.630+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:21:54.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.630+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:54.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.630+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:21:54.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.630+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:54.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.630+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:54.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.634+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:54.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.634+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:54.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.634+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:54.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.634+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:54.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.634+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800bb60 con 0x7f833801bab0 2021-03-09T09:21:54.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.634+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334004f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:54.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f833801bab0 2021-03-09T09:21:54.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833406ddd0 con 0x7f833801bab0 2021-03-09T09:21:54.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:54.638+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833405fa20 con 0x7f833801bab0 2021-03-09T09:21:55.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:21:54 smithi146 bash[13136]: debug 2021-03-09T09:21:54.626+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5755 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:21:59.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.630+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:21:59.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.630+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:59.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.630+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334004f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:59.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.630+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334004f50).stop 2021-03-09T09:21:59.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:59.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.630+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:59.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.630+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:59.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.630+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:59.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.630+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:59.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f834c0f4690 2021-03-09T09:21:59.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014ee0 con 0x7f8338026c80 2021-03-09T09:21:59.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380122e0 con 0x7f8338062ca0 2021-03-09T09:21:59.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.630+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:21:59.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.630+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:59.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.630+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:21:59.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.630+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:59.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.634+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:59.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.634+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:59.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.634+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:59.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.634+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:59.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.634+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:59.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.634+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010060 con 0x7f8338026c80 2021-03-09T09:21:59.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.638+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380701d0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 rx=0x7f833404e7a0 tx=0x7f8334004f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:59.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.638+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f8338026c80 2021-03-09T09:21:59.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.638+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404e5e0 con 0x7f8338026c80 2021-03-09T09:21:59.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:21:59.638+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334064220 con 0x7f8338026c80 2021-03-09T09:22:00.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:21:59 smithi146 bash[13136]: debug 2021-03-09T09:21:59.630+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5779 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:22:04.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f83380701d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:22:04.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f83380701d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:04.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380701d0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 rx=0x7f833404e7a0 tx=0x7f8334004f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:04.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380701d0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 rx=0x7f833404e7a0 tx=0x7f8334004f50).stop 2021-03-09T09:22:04.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f83380701d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:04.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380701d0 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:04.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:04.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:04.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:04.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f834c0f4690 2021-03-09T09:22:04.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806b260 con 0x7f833801bab0 2021-03-09T09:22:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010ab0 con 0x7f8338062ca0 2021-03-09T09:22:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:22:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:22:04.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:04.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.630+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:04.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.634+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:04.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.634+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:04.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.634+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:04.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.634+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:04.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.634+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338027940 con 0x7f833801bab0 2021-03-09T09:22:04.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.634+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334028460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:04.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334047460 con 0x7f833801bab0 2021-03-09T09:22:04.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334031540 con 0x7f833801bab0 2021-03-09T09:22:04.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:04.638+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334033cc0 con 0x7f833801bab0 2021-03-09T09:22:05.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:22:04 smithi146 bash[13136]: debug 2021-03-09T09:22:04.630+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5805 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:22:09.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.630+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:22:09.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.630+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:09.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.630+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334028460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:09.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.630+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334028460).stop 2021-03-09T09:22:09.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.630+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:09.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.634+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:09.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.634+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:09.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.634+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380163f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:09.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.634+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:09.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d170 con 0x7f834c0f4690 2021-03-09T09:22:09.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f8338026c80 2021-03-09T09:22:09.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005990 con 0x7f8338062ca0 2021-03-09T09:22:09.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.634+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:22:09.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.634+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:09.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.634+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:22:09.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.634+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:09.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.634+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380163f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:09.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.638+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:09.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.638+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.638+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.638+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.638+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338069e00 con 0x7f8338026c80 2021-03-09T09:22:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.638+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380163f0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f834400c1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:09.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.638+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f8338026c80 2021-03-09T09:22:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.642+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338026c80 2021-03-09T09:22:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:09.642+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404ad20 con 0x7f8338026c80 2021-03-09T09:22:10.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:22:09 smithi146 bash[13136]: debug 2021-03-09T09:22:09.634+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5825 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:22:14.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f83380163f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:22:14.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f83380163f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:14.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380163f0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f834400c1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:14.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380163f0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f834400c1f0).stop 2021-03-09T09:22:14.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f83380163f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:14.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380163f0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:14.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:14.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066610 con 0x7f834c0f4690 2021-03-09T09:22:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f833801bab0 2021-03-09T09:22:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065e40 con 0x7f8338062ca0 2021-03-09T09:22:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:22:14.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:22:14.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:14.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:14.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.634+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:14.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.638+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.638+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.638+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.638+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.638+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800cff0 con 0x7f833801bab0 2021-03-09T09:22:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.638+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834400c1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:14.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.638+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344016580 con 0x7f833801bab0 2021-03-09T09:22:14.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.642+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404ad20 con 0x7f833801bab0 2021-03-09T09:22:14.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:14.646+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404ad20 con 0x7f833801bab0 2021-03-09T09:22:15.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:22:14 smithi146 bash[13136]: debug 2021-03-09T09:22:14.634+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5855 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:22:19.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:22:19.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:19.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834400c1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:19.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834400c1f0).stop 2021-03-09T09:22:19.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:19.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:19.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:19.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:19.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:19.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012b40 con 0x7f834c0f4690 2021-03-09T09:22:19.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010ab0 con 0x7f8338026c80 2021-03-09T09:22:19.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014ee0 con 0x7f8338062ca0 2021-03-09T09:22:19.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:22:19.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:19.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:22:19.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:19.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.634+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:19.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.638+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:19.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.638+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:19.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.638+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:19.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.638+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:19.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.638+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026710 con 0x7f8338026c80 2021-03-09T09:22:19.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.638+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800ed40 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 rx=0x7f833403a390 tx=0x7f8334028460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:19.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.638+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f8338026c80 2021-03-09T09:22:19.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.638+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8338026c80 2021-03-09T09:22:19.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:19.642+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334031130 con 0x7f8338026c80 2021-03-09T09:22:20.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:22:19 smithi146 bash[13136]: debug 2021-03-09T09:22:19.634+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5879 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:22:24.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:22:24.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:24.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800ed40 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 rx=0x7f833403a390 tx=0x7f8334028460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:24.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800ed40 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 rx=0x7f833403a390 tx=0x7f8334028460).stop 2021-03-09T09:22:24.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800ed40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:24.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800ed40 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:24.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004d60 con 0x7f834c0f4690 2021-03-09T09:22:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065e40 con 0x7f833801bab0 2021-03-09T09:22:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014ee0 con 0x7f8338062ca0 2021-03-09T09:22:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:22:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:22:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:24.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.637+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380114e0 con 0x7f833801bab0 2021-03-09T09:22:24.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.641+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 rx=0x7f833c0299a0 tx=0x7f833c01d000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:24.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0574c0 con 0x7f833801bab0 2021-03-09T09:22:24.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f833801bab0 2021-03-09T09:22:24.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:24.649+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f833801bab0 2021-03-09T09:22:25.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:22:24 smithi146 bash[13136]: debug 2021-03-09T09:22:24.637+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5904 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:22:29.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:22:29.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:29.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 rx=0x7f833c0299a0 tx=0x7f833c01d000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:29.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 rx=0x7f833c0299a0 tx=0x7f833c01d000).stop 2021-03-09T09:22:29.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:29.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:29.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:29.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:29.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:29.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801fde0 con 0x7f834c0f4690 2021-03-09T09:22:29.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f8338026c80 2021-03-09T09:22:29.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e470 con 0x7f8338062ca0 2021-03-09T09:22:29.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:22:29.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:29.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:22:29.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:29.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.637+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:29.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.641+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:29.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.641+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:29.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.641+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:29.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.641+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:29.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.641+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064e60 con 0x7f8338026c80 2021-03-09T09:22:29.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.641+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f5f0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834400c1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:29.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.641+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029040 con 0x7f8338026c80 2021-03-09T09:22:29.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.641+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402b760 con 0x7f8338026c80 2021-03-09T09:22:29.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:29.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400d450 con 0x7f8338026c80 2021-03-09T09:22:30.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:22:29 smithi146 bash[13136]: debug 2021-03-09T09:22:29.637+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5929 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:22:34.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:22:34.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:34.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f5f0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834400c1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:34.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f5f0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834400c1f0).stop 2021-03-09T09:22:34.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800f5f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:34.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f5f0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:34.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:34.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:34.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:34.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e590 con 0x7f834c0f4690 2021-03-09T09:22:34.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338070ab0 con 0x7f833801bab0 2021-03-09T09:22:34.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012180 con 0x7f8338062ca0 2021-03-09T09:22:34.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:22:34.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:34.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:22:34.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:34.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:34.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:34.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:34.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:34.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:34.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.641+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338018060 con 0x7f833801bab0 2021-03-09T09:22:34.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.645+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 rx=0x7f8344027df0 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:34.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029040 con 0x7f833801bab0 2021-03-09T09:22:34.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f833801bab0 2021-03-09T09:22:34.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:34.649+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344010c20 con 0x7f833801bab0 2021-03-09T09:22:35.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:22:34 smithi146 bash[13136]: debug 2021-03-09T09:22:34.641+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5954 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:22:39.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:22:39.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:39.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 rx=0x7f8344027df0 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:39.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 rx=0x7f8344027df0 tx=0x7f8344021000).stop 2021-03-09T09:22:39.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:39.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:39.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:39.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:22:39.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:39.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:39.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338017720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:39.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020630 con 0x7f834c0f4690 2021-03-09T09:22:39.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e470 con 0x7f8338026c80 2021-03-09T09:22:39.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380145f0 con 0x7f8338062ca0 2021-03-09T09:22:39.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:22:39.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:39.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.641+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338017720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:39.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:39.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:39.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:39.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:39.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017c10 con 0x7f8338026c80 2021-03-09T09:22:39.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338017720 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 rx=0x7f833c053840 tx=0x7f833c01d000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:39.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f8338026c80 2021-03-09T09:22:39.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f8338026c80 2021-03-09T09:22:39.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:39.649+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004070 con 0x7f8338026c80 2021-03-09T09:22:40.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:22:39 smithi146 bash[13136]: debug 2021-03-09T09:22:39.641+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 5977 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:22:44.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338017720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:22:44.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338017720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:44.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338017720 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 rx=0x7f833c053840 tx=0x7f833c01d000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:44.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338017720 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 rx=0x7f833c053840 tx=0x7f833c01d000).stop 2021-03-09T09:22:44.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338017720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:44.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338017720 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:44.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:44.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:44.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:44.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380687c0 con 0x7f834c0f4690 2021-03-09T09:22:44.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020630 con 0x7f833801bab0 2021-03-09T09:22:44.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012b40 con 0x7f8338062ca0 2021-03-09T09:22:44.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:22:44.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:22:44.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:44.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:44.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:44.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:44.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:44.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:44.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:44.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338023d00 con 0x7f833801bab0 2021-03-09T09:22:44.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c009eb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:44.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00c360 con 0x7f833801bab0 2021-03-09T09:22:44.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833801bab0 2021-03-09T09:22:44.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:44.649+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c022730 con 0x7f833801bab0 2021-03-09T09:22:45.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:22:44 smithi146 bash[13136]: debug 2021-03-09T09:22:44.641+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6004 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:22:49.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:22:49.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:49.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c009eb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:49.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c009eb0).stop 2021-03-09T09:22:49.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:49.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:49.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:49.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:49.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:49.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020630 con 0x7f834c0f4690 2021-03-09T09:22:49.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dae0 con 0x7f8338026c80 2021-03-09T09:22:49.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a860 con 0x7f8338062ca0 2021-03-09T09:22:49.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:22:49.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:49.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:22:49.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:49.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:49.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:49.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.645+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:49.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.649+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:49.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.649+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:49.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.649+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380145a0 con 0x7f8338026c80 2021-03-09T09:22:49.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.649+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f5f0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:49.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.649+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029040 con 0x7f8338026c80 2021-03-09T09:22:49.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.649+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400e270 con 0x7f8338026c80 2021-03-09T09:22:49.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:49.653+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402b220 con 0x7f8338026c80 2021-03-09T09:22:50.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:22:49 smithi146 bash[13136]: debug 2021-03-09T09:22:49.641+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6029 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:22:54.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:22:54.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:54.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f5f0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:54.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f5f0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344021000).stop 2021-03-09T09:22:54.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800f5f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:54.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f5f0 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:54.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:54.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:54.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:54.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020630 con 0x7f834c0f4690 2021-03-09T09:22:54.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c090 con 0x7f833801bab0 2021-03-09T09:22:54.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010ab0 con 0x7f8338062ca0 2021-03-09T09:22:54.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:22:54.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:54.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:22:54.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:54.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:54.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:54.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:54.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:54.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:54.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022dc0 con 0x7f833801bab0 2021-03-09T09:22:54.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 rx=0x7f833c0128d0 tx=0x7f833c036000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:54.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f833801bab0 2021-03-09T09:22:54.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.649+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f833801bab0 2021-03-09T09:22:54.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:54.653+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c077bf0 con 0x7f833801bab0 2021-03-09T09:22:55.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:22:54 smithi146 bash[13136]: debug 2021-03-09T09:22:54.645+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6054 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:22:59.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:22:59.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:59.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 rx=0x7f833c0128d0 tx=0x7f833c036000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:59.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 rx=0x7f833c0128d0 tx=0x7f833c036000).stop 2021-03-09T09:22:59.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:59.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.645+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:59.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.645+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:59.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.645+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:59.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.645+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:59.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020630 con 0x7f834c0f4690 2021-03-09T09:22:59.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019f30 con 0x7f8338026c80 2021-03-09T09:22:59.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8338062ca0 2021-03-09T09:22:59.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:22:59.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:59.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.645+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:22:59.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.649+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:59.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.649+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:59.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.649+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:59.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.649+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:59.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.649+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:59.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.649+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:59.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.649+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024f20 con 0x7f8338026c80 2021-03-09T09:22:59.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.649+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800fd00 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 rx=0x7f834401f2e0 tx=0x7f834406b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:59.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.653+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f8338026c80 2021-03-09T09:22:59.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.653+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834401f120 con 0x7f8338026c80 2021-03-09T09:22:59.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:22:59.657+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338026c80 2021-03-09T09:23:00.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:22:59 smithi146 bash[13136]: debug 2021-03-09T09:22:59.649+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6079 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:23:04.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:23:04.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:04.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800fd00 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 rx=0x7f834401f2e0 tx=0x7f834406b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:04.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800fd00 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 rx=0x7f834401f2e0 tx=0x7f834406b000).stop 2021-03-09T09:23:04.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800fd00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:04.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800fd00 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024de0 con 0x7f834c0f4690 2021-03-09T09:23:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cf20 con 0x7f833801bab0 2021-03-09T09:23:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016630 con 0x7f8338062ca0 2021-03-09T09:23:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:23:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:23:04.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:04.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.648+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:04.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.652+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:04.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.652+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:04.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.652+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:04.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.652+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:04.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.652+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026940 con 0x7f833801bab0 2021-03-09T09:23:04.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.652+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 rx=0x7f83440168e0 tx=0x7f834406b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:04.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029030 con 0x7f833801bab0 2021-03-09T09:23:04.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402b760 con 0x7f833801bab0 2021-03-09T09:23:04.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:04.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833801bab0 2021-03-09T09:23:05.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:23:04 smithi146 bash[13136]: debug 2021-03-09T09:23:04.649+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6104 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:23:09.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:23:09.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:09.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 rx=0x7f83440168e0 tx=0x7f834406b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 rx=0x7f83440168e0 tx=0x7f834406b000).stop 2021-03-09T09:23:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338025110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020630 con 0x7f834c0f4690 2021-03-09T09:23:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012b40 con 0x7f8338026c80 2021-03-09T09:23:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ede0 con 0x7f8338062ca0 2021-03-09T09:23:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:23:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:23:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.648+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338025110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.652+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.652+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.652+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.652+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.652+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022dc0 con 0x7f8338026c80 2021-03-09T09:23:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.652+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338025110 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c036000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01a270 con 0x7f8338026c80 2021-03-09T09:23:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8338026c80 2021-03-09T09:23:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:09.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338026c80 2021-03-09T09:23:10.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:23:09 smithi146 bash[13136]: debug 2021-03-09T09:23:09.649+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6129 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:23:14.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338025110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:23:14.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338025110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:14.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338025110 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c036000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:14.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338025110 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c036000).stop 2021-03-09T09:23:14.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338025110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:14.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338025110 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:14.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:14.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:14.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:14.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020630 con 0x7f834c0f4690 2021-03-09T09:23:14.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b740 con 0x7f833801bab0 2021-03-09T09:23:14.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ede0 con 0x7f8338062ca0 2021-03-09T09:23:14.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:23:14.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:14.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:23:14.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:14.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:14.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:14.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:14.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:14.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:14.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026940 con 0x7f833801bab0 2021-03-09T09:23:14.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 rx=0x7f83340649d0 tx=0x7f8334028460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:14.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340647d0 con 0x7f833801bab0 2021-03-09T09:23:14.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334064400 con 0x7f833801bab0 2021-03-09T09:23:14.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:14.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833400bc20 con 0x7f833801bab0 2021-03-09T09:23:15.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:23:14 smithi146 bash[13136]: debug 2021-03-09T09:23:14.649+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6153 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:23:19.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:23:19.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:19.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 rx=0x7f83340649d0 tx=0x7f8334028460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:19.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 rx=0x7f83340649d0 tx=0x7f8334028460).stop 2021-03-09T09:23:19.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:19.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:19.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:19.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:19.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:19.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380093b0 con 0x7f834c0f4690 2021-03-09T09:23:19.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020630 con 0x7f8338026c80 2021-03-09T09:23:19.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026760 con 0x7f8338062ca0 2021-03-09T09:23:19.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:23:19.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:19.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:23:19.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:19.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.652+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:19.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.656+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:19.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.656+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:19.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.656+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:19.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.656+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:19.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.656+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022dc0 con 0x7f8338026c80 2021-03-09T09:23:19.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.656+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338069980 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834406b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:19.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f8338026c80 2021-03-09T09:23:19.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402b760 con 0x7f8338026c80 2021-03-09T09:23:19.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:19.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402b220 con 0x7f8338026c80 2021-03-09T09:23:20.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:23:19 smithi146 bash[13136]: debug 2021-03-09T09:23:19.652+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6179 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:23:24.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:23:24.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:24.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338069980 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834406b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:24.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338069980 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834406b000).stop 2021-03-09T09:23:24.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338069980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:24.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338069980 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:24.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:24.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:24.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:24.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019f30 con 0x7f834c0f4690 2021-03-09T09:23:24.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012b40 con 0x7f833801bab0 2021-03-09T09:23:24.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ede0 con 0x7f8338062ca0 2021-03-09T09:23:24.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:23:24.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:24.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:23:24.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:24.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:24.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:24.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:24.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:24.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:24.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026940 con 0x7f833801bab0 2021-03-09T09:23:24.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7f833c053750 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:24.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833801bab0 2021-03-09T09:23:24.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c053af0 con 0x7f833801bab0 2021-03-09T09:23:24.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:24.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05f850 con 0x7f833801bab0 2021-03-09T09:23:25.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:23:24 smithi146 bash[13136]: debug 2021-03-09T09:23:24.656+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6204 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:23:29.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:23:29.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:29.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7f833c053750 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:29.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7f833c053750 tx=0x7f833c00b040).stop 2021-03-09T09:23:29.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:29.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:29.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:29.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:29.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:29.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b8d0 con 0x7f834c0f4690 2021-03-09T09:23:29.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071300 con 0x7f8338026c80 2021-03-09T09:23:29.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806f020 con 0x7f8338062ca0 2021-03-09T09:23:29.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:23:29.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:29.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:23:29.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:29.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:29.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:29.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:29.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:29.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:29.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338021cf0 con 0x7f8338026c80 2021-03-09T09:23:29.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.660+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 rx=0x7f833c04e4f0 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:29.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338026c80 2021-03-09T09:23:29.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f8338026c80 2021-03-09T09:23:29.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:29.664+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c053270 con 0x7f8338026c80 2021-03-09T09:23:30.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:23:29 smithi146 bash[13136]: debug 2021-03-09T09:23:29.656+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6228 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:23:34.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:23:34.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:34.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 rx=0x7f833c04e4f0 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:34.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 rx=0x7f833c04e4f0 tx=0x7f833c00b040).stop 2021-03-09T09:23:34.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:34.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:34.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:34.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:34.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:34.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b8d0 con 0x7f834c0f4690 2021-03-09T09:23:34.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012b40 con 0x7f833801bab0 2021-03-09T09:23:34.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802df60 con 0x7f8338062ca0 2021-03-09T09:23:34.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:34.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:23:34.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:34.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:23:34.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:34.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:34.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:34.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:34.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:34.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833805eb00 con 0x7f833801bab0 2021-03-09T09:23:34.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 rx=0x7f8334024c20 tx=0x7f8334028460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:34.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f833801bab0 2021-03-09T09:23:34.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007c30 con 0x7f833801bab0 2021-03-09T09:23:34.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:34.664+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d4b0 con 0x7f833801bab0 2021-03-09T09:23:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:23:34 smithi146 bash[13136]: debug 2021-03-09T09:23:34.656+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6254 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:23:39.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:23:39.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:39.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 rx=0x7f8334024c20 tx=0x7f8334028460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:39.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 rx=0x7f8334024c20 tx=0x7f8334028460).stop 2021-03-09T09:23:39.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:39.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:39.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:39.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:39.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:39.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802df60 con 0x7f834c0f4690 2021-03-09T09:23:39.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806fcc0 con 0x7f8338026c80 2021-03-09T09:23:39.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071310 con 0x7f8338062ca0 2021-03-09T09:23:39.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:23:39.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:39.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:23:39.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:39.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:39.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:39.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:39.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:39.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:39.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.660+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022dc0 con 0x7f8338026c80 2021-03-09T09:23:39.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.664+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7f8338066da0 tx=0x7f834406b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:39.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.668+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f8338026c80 2021-03-09T09:23:39.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.668+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f8338026c80 2021-03-09T09:23:39.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:39.668+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338026c80 2021-03-09T09:23:40.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:23:39 smithi146 bash[13136]: debug 2021-03-09T09:23:39.656+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6278 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:23:44.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:23:44.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:44.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7f8338066da0 tx=0x7f834406b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:44.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7f8338066da0 tx=0x7f834406b000).stop 2021-03-09T09:23:44.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800b620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:44.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:44.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:44.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:44.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:44.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f834c0f4690 2021-03-09T09:23:44.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ffe0 con 0x7f833801bab0 2021-03-09T09:23:44.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ad60 con 0x7f8338062ca0 2021-03-09T09:23:44.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:23:44.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:23:44.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:44.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:44.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:44.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:44.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:44.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:44.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:44.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806d320 con 0x7f833801bab0 2021-03-09T09:23:44.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834406b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:44.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f833801bab0 2021-03-09T09:23:44.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.664+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f833801bab0 2021-03-09T09:23:44.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:44.668+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404b1d0 con 0x7f833801bab0 2021-03-09T09:23:45.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:23:44 smithi146 bash[13136]: debug 2021-03-09T09:23:44.660+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6302 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:23:49.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:23:49.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:49.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834406b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:49.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834406b000).stop 2021-03-09T09:23:49.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:49.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:49.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:49.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:49.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833801ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:49.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f834c0f4690 2021-03-09T09:23:49.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ffe0 con 0x7f8338026c80 2021-03-09T09:23:49.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065ff0 con 0x7f8338062ca0 2021-03-09T09:23:49.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:23:49.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:49.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:23:49.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:49.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833801ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:49.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.667+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:49.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:49.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.667+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:49.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:49.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.667+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012af0 con 0x7f8338026c80 2021-03-09T09:23:49.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833801ad60 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 rx=0x7f833c06aae0 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:49.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f8338026c80 2021-03-09T09:23:49.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f8338026c80 2021-03-09T09:23:49.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:49.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01a270 con 0x7f8338026c80 2021-03-09T09:23:50.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:23:49 smithi146 bash[13136]: debug 2021-03-09T09:23:49.664+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6328 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:23:54.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833801ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:23:54.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833801ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:54.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833801ad60 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 rx=0x7f833c06aae0 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:54.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833801ad60 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 rx=0x7f833c06aae0 tx=0x7f833c00b040).stop 2021-03-09T09:23:54.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833801ad60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:54.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833801ad60 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:54.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:54.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:54.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:54.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f834c0f4690 2021-03-09T09:23:54.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ffe0 con 0x7f833801bab0 2021-03-09T09:23:54.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071310 con 0x7f8338062ca0 2021-03-09T09:23:54.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.663+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:23:54.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.663+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:54.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.667+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:23:54.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.667+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:54.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:54.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.667+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:54.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:54.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.667+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:54.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:54.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.667+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380058e0 con 0x7f833801bab0 2021-03-09T09:23:54.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 rx=0x7f833c077dd0 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:54.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c077050 con 0x7f833801bab0 2021-03-09T09:23:54.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f833801bab0 2021-03-09T09:23:54.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:54.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06ad10 con 0x7f833801bab0 2021-03-09T09:23:55.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:23:54 smithi146 bash[13136]: debug 2021-03-09T09:23:54.664+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6354 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:23:59.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:23:59.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:59.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 rx=0x7f833c077dd0 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:59.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 rx=0x7f833c077dd0 tx=0x7f833c00b040).stop 2021-03-09T09:23:59.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:59.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:59.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:59.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:59.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:59.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802df60 con 0x7f834c0f4690 2021-03-09T09:23:59.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066d20 con 0x7f8338026c80 2021-03-09T09:23:59.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b160 con 0x7f8338062ca0 2021-03-09T09:23:59.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:23:59.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:59.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:23:59.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:59.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.667+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:59.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:59.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:59.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:59.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:59.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380688c0 con 0x7f8338026c80 2021-03-09T09:23:59.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 rx=0x7f834401fdd0 tx=0x7f834406b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:59.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f8338026c80 2021-03-09T09:23:59.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402b760 con 0x7f8338026c80 2021-03-09T09:23:59.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:23:59.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404fd20 con 0x7f8338026c80 2021-03-09T09:24:00.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:23:59 smithi146 bash[13136]: debug 2021-03-09T09:23:59.664+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6377 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:24:04.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:24:04.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:04.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 rx=0x7f834401fdd0 tx=0x7f834406b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:04.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 rx=0x7f834401fdd0 tx=0x7f834406b000).stop 2021-03-09T09:24:04.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800b620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:04.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:04.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:04.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:04.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b160 con 0x7f834c0f4690 2021-03-09T09:24:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016830 con 0x7f833801bab0 2021-03-09T09:24:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071310 con 0x7f8338062ca0 2021-03-09T09:24:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:24:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:24:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.667+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.671+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:04.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.671+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:04.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.671+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:04.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.671+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:04.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.671+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065440 con 0x7f833801bab0 2021-03-09T09:24:04.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.671+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 rx=0x7f8334000c30 tx=0x7f8334028460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:04.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032030 con 0x7f833801bab0 2021-03-09T09:24:04.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007c30 con 0x7f833801bab0 2021-03-09T09:24:04.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:04.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401f6f0 con 0x7f833801bab0 2021-03-09T09:24:05.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:24:04 smithi146 bash[13136]: debug 2021-03-09T09:24:04.668+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6403 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:24:09.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.667+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:24:09.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.667+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:09.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.667+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 rx=0x7f8334000c30 tx=0x7f8334028460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:09.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.667+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 rx=0x7f8334000c30 tx=0x7f8334028460).stop 2021-03-09T09:24:09.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:09.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:09.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:09.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:09.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:09.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb30 con 0x7f834c0f4690 2021-03-09T09:24:09.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802df60 con 0x7f8338026c80 2021-03-09T09:24:09.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b160 con 0x7f8338062ca0 2021-03-09T09:24:09.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:24:09.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:24:09.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:09.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:09.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:09.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:09.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:09.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:09.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:09.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380688c0 con 0x7f8338026c80 2021-03-09T09:24:09.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 rx=0x7f8344072300 tx=0x7f834406b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:09.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402b760 con 0x7f8338026c80 2021-03-09T09:24:09.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338026c80 2021-03-09T09:24:09.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:09.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338026c80 2021-03-09T09:24:10.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:24:09 smithi146 bash[13136]: debug 2021-03-09T09:24:09.668+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6424 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:24:14.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:24:14.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:14.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 rx=0x7f8344072300 tx=0x7f834406b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:14.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 rx=0x7f8344072300 tx=0x7f834406b000).stop 2021-03-09T09:24:14.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800b620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:14.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800b620 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:14.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:14.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:14.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:14.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380691c0 con 0x7f834c0f4690 2021-03-09T09:24:14.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb30 con 0x7f833801bab0 2021-03-09T09:24:14.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071310 con 0x7f8338062ca0 2021-03-09T09:24:14.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:24:14.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:14.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:24:14.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:14.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:14.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:14.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:14.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:14.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:14.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.671+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380259d0 con 0x7f833801bab0 2021-03-09T09:24:14.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.675+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801b070 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 rx=0x7f833406d570 tx=0x7f8334028460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:14.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f833801bab0 2021-03-09T09:24:14.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402d6a0 con 0x7f833801bab0 2021-03-09T09:24:14.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:14.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334024000 con 0x7f833801bab0 2021-03-09T09:24:15.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:24:14 smithi146 bash[13136]: debug 2021-03-09T09:24:14.672+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6450 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:24:19.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.671+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:24:19.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.671+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:19.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.671+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801b070 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 rx=0x7f833406d570 tx=0x7f8334028460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:19.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.671+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801b070 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 rx=0x7f833406d570 tx=0x7f8334028460).stop 2021-03-09T09:24:19.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801b070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:19.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801b070 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:19.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:19.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:19.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338070130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:19.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801fe80 con 0x7f834c0f4690 2021-03-09T09:24:19.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb30 con 0x7f8338026c80 2021-03-09T09:24:19.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a990 con 0x7f8338062ca0 2021-03-09T09:24:19.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:24:19.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:19.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.675+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338070130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:19.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.675+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:24:19.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.675+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:19.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.675+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:19.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.675+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:19.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.675+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:19.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.675+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:19.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.675+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065440 con 0x7f8338026c80 2021-03-09T09:24:19.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.675+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338070130 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 rx=0x7f833c01a5b0 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:19.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c077bf0 con 0x7f8338026c80 2021-03-09T09:24:19.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f8338026c80 2021-03-09T09:24:19.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:19.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338026c80 2021-03-09T09:24:20.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:24:19 smithi146 bash[13136]: debug 2021-03-09T09:24:19.671+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6476 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:24:24.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338070130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:24:24.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338070130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:24.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338070130 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 rx=0x7f833c01a5b0 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:24.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338070130 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 rx=0x7f833c01a5b0 tx=0x7f833c00b040).stop 2021-03-09T09:24:24.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338070130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:24.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338070130 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:24.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:24.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:24.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:24.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071310 con 0x7f834c0f4690 2021-03-09T09:24:24.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026ad0 con 0x7f833801bab0 2021-03-09T09:24:24.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801fd40 con 0x7f8338062ca0 2021-03-09T09:24:24.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:24:24.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:24.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:24:24.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:24.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.675+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:24.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.679+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f83380630a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:24.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.679+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f83380630a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:24.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.679+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:24.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.679+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:24.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.679+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338067620 con 0x7f833801bab0 2021-03-09T09:24:24.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.679+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834406b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:24.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833801bab0 2021-03-09T09:24:24.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402b760 con 0x7f833801bab0 2021-03-09T09:24:24.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:24.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833801bab0 2021-03-09T09:24:25.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:24:24 smithi146 bash[13136]: debug 2021-03-09T09:24:24.675+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6503 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:24:29.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:24:29.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:29.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834406b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:29.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834406b000).stop 2021-03-09T09:24:29.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:29.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:29.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:29.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:29.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:29.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:24:29.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:29.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:24:29.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:29.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f834c0f4690 2021-03-09T09:24:29.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380152f0 con 0x7f8338026c80 2021-03-09T09:24:29.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022080 con 0x7f8338062ca0 2021-03-09T09:24:29.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.674+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:29.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.678+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:29.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.678+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:29.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.678+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:29.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.678+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:29.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.678+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064e60 con 0x7f8338026c80 2021-03-09T09:24:29.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.678+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802d310 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 rx=0x7f8334026dd0 tx=0x7f833404e1e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:29.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.678+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f8338026c80 2021-03-09T09:24:29.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.678+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334052450 con 0x7f8338026c80 2021-03-09T09:24:29.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:29.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403a9e0 con 0x7f8338026c80 2021-03-09T09:24:30.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:24:29 smithi146 bash[13136]: debug 2021-03-09T09:24:29.675+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6524 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:24:34.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833802d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:24:34.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833802d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:34.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802d310 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 rx=0x7f8334026dd0 tx=0x7f833404e1e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:34.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802d310 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 rx=0x7f8334026dd0 tx=0x7f833404e1e0).stop 2021-03-09T09:24:34.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833802d310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:34.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802d310 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:34.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f83380661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:34.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:34.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:34.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:24:34.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:34.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb30 con 0x7f834c0f4690 2021-03-09T09:24:34.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ede0 con 0x7f833801bab0 2021-03-09T09:24:34.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a990 con 0x7f8338062ca0 2021-03-09T09:24:34.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:24:34.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:34.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f83380661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:34.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:34.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:34.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:34.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:34.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380144e0 con 0x7f833801bab0 2021-03-09T09:24:34.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.678+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f83380661c0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f833404e1e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:34.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833403abf0 con 0x7f833801bab0 2021-03-09T09:24:34.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403abf0 con 0x7f833801bab0 2021-03-09T09:24:34.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:34.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403a3a0 con 0x7f833801bab0 2021-03-09T09:24:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:24:34 smithi146 bash[13136]: debug 2021-03-09T09:24:34.675+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6553 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:24:39.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f83380661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:24:39.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f83380661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:39.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f83380661c0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f833404e1e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:39.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f83380661c0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f833404e1e0).stop 2021-03-09T09:24:39.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f83380661c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:39.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f83380661c0 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:39.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:39.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:39.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:24:39.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:39.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:39.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071310 con 0x7f834c0f4690 2021-03-09T09:24:39.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065240 con 0x7f8338026c80 2021-03-09T09:24:39.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380152f0 con 0x7f8338062ca0 2021-03-09T09:24:39.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:24:39.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:39.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.678+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:39.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.682+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:39.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.682+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:39.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.682+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:39.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.682+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:39.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.682+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020ca0 con 0x7f8338026c80 2021-03-09T09:24:39.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.682+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338021080 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 rx=0x7f8338068860 tx=0x7f834406b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:39.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029030 con 0x7f8338026c80 2021-03-09T09:24:39.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338026c80 2021-03-09T09:24:39.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:39.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404ad20 con 0x7f8338026c80 2021-03-09T09:24:40.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:24:39 smithi146 bash[13136]: debug 2021-03-09T09:24:39.679+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6576 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:24:44.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.678+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:24:44.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.678+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:44.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.678+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338021080 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 rx=0x7f8338068860 tx=0x7f834406b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:44.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.678+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338021080 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 rx=0x7f8338068860 tx=0x7f834406b000).stop 2021-03-09T09:24:44.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.678+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:44.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.678+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338021080 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:44.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.678+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:44.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.678+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:44.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.678+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:44.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.678+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:24:44.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.678+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:44.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.678+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:24:44.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.678+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:44.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022080 con 0x7f834c0f4690 2021-03-09T09:24:44.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018b10 con 0x7f833801bab0 2021-03-09T09:24:44.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb30 con 0x7f8338062ca0 2021-03-09T09:24:44.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:44.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.682+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:44.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:44.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.682+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:44.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:44.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.682+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801a940 con 0x7f833801bab0 2021-03-09T09:24:44.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 rx=0x7f833c04d4a0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:44.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f833801bab0 2021-03-09T09:24:44.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833801bab0 2021-03-09T09:24:44.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:44.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c310 con 0x7f833801bab0 2021-03-09T09:24:45.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:24:44 smithi146 bash[13136]: debug 2021-03-09T09:24:44.679+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6603 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:24:49.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:24:50.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:50.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 rx=0x7f833c04d4a0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:50.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 rx=0x7f833c04d4a0 tx=0x7f833c04e870).stop 2021-03-09T09:24:50.075 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:24:49 smithi146 bash[13136]: debug 2021-03-09T09:24:49.683+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6626 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:24:50.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:50.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:50.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:50.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:50.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:50.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380691c0 con 0x7f834c0f4690 2021-03-09T09:24:50.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:24:50.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:24:50.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:50.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:50.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f8338026c80 2021-03-09T09:24:50.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eea0 con 0x7f8338062ca0 2021-03-09T09:24:50.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.682+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:50.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.686+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:50.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.686+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:50.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.686+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:50.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.686+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:50.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.686+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338067620 con 0x7f8338026c80 2021-03-09T09:24:50.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.686+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 rx=0x7f833403b040 tx=0x7f833404e1e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:50.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338026c80 2021-03-09T09:24:50.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000d20 con 0x7f8338026c80 2021-03-09T09:24:50.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:49.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334031130 con 0x7f8338026c80 2021-03-09T09:24:54.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.682+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:24:54.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.682+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:54.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.682+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 rx=0x7f833403b040 tx=0x7f833404e1e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:54.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.682+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 rx=0x7f833403b040 tx=0x7f833404e1e0).stop 2021-03-09T09:24:54.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338063080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:54.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:54.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:54.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:54.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:54.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c950 con 0x7f834c0f4690 2021-03-09T09:24:54.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb30 con 0x7f833801bab0 2021-03-09T09:24:54.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024ca0 con 0x7f8338062ca0 2021-03-09T09:24:54.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.682+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:24:54.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.682+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:24:54.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.682+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:54.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:54.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.686+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:54.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.686+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:54.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.686+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:54.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.686+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:54.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.686+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:54.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.686+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338067620 con 0x7f833801bab0 2021-03-09T09:24:54.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.686+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834406b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:54.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029040 con 0x7f833801bab0 2021-03-09T09:24:54.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072190 con 0x7f833801bab0 2021-03-09T09:24:54.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:54.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833801bab0 2021-03-09T09:24:55.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:24:54 smithi146 bash[13136]: debug 2021-03-09T09:24:54.683+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6649 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:24:59.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.682+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:24:59.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.682+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:59.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.682+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834406b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:59.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.682+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834406b000).stop 2021-03-09T09:24:59.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:59.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.686+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:59.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.686+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:59.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.686+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:59.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.686+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:24:59.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.686+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:59.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.686+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:59.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.686+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:24:59.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ede0 con 0x7f834c0f4690 2021-03-09T09:24:59.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380284b0 con 0x7f8338026c80 2021-03-09T09:24:59.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c760 con 0x7f8338062ca0 2021-03-09T09:24:59.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.686+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:59.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.686+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:59.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.686+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:59.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.686+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:59.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.690+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:59.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.690+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:59.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.690+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806cf30 con 0x7f8338026c80 2021-03-09T09:24:59.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.690+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 rx=0x7f834401f6d0 tx=0x7f83440045f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:59.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029030 con 0x7f8338026c80 2021-03-09T09:24:59.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834401f300 con 0x7f8338026c80 2021-03-09T09:24:59.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:24:59.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834406a0e0 con 0x7f8338026c80 2021-03-09T09:25:00.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:24:59 smithi146 bash[13136]: debug 2021-03-09T09:24:59.683+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6676 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:25:04.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:25:04.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:04.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 rx=0x7f834401f6d0 tx=0x7f83440045f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:04.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 rx=0x7f834401f6d0 tx=0x7f83440045f0).stop 2021-03-09T09:25:04.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338063080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:04.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:04.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:04.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:04.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:04.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:25:04.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:04.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:25:04.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:04.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802dd60 con 0x7f834c0f4690 2021-03-09T09:25:04.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806ffd0 con 0x7f833801bab0 2021-03-09T09:25:04.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024ca0 con 0x7f8338062ca0 2021-03-09T09:25:04.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.686+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:04.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.690+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:04.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.690+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:04.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.690+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:04.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.690+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:04.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.690+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338069170 con 0x7f833801bab0 2021-03-09T09:25:04.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.690+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 rx=0x7f8334063da0 tx=0x7f833404e1e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:04.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f833801bab0 2021-03-09T09:25:04.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334063dd0 con 0x7f833801bab0 2021-03-09T09:25:04.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:04.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d300 con 0x7f833801bab0 2021-03-09T09:25:05.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:25:04 smithi146 bash[13136]: debug 2021-03-09T09:25:04.687+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6703 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:25:09.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:25:09.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:09.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 rx=0x7f8334063da0 tx=0x7f833404e1e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:09.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 rx=0x7f8334063da0 tx=0x7f833404e1e0).stop 2021-03-09T09:25:09.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:09.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:09.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:09.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:09.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:09.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:25:09.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:09.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:25:09.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:09.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062520 con 0x7f834c0f4690 2021-03-09T09:25:09.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067060 con 0x7f8338026c80 2021-03-09T09:25:09.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c760 con 0x7f8338062ca0 2021-03-09T09:25:09.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:09.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:09.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:09.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:09.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:09.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.690+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380152a0 con 0x7f8338026c80 2021-03-09T09:25:09.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.694+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f833404e1e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:09.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340248d0 con 0x7f8338026c80 2021-03-09T09:25:09.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403be20 con 0x7f8338026c80 2021-03-09T09:25:09.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:09.698+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340643e0 con 0x7f8338026c80 2021-03-09T09:25:10.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:25:09 smithi146 bash[13136]: debug 2021-03-09T09:25:09.687+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6726 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:25:14.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:25:14.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:14.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f833404e1e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:14.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f833404e1e0).stop 2021-03-09T09:25:14.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338063080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:14.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:14.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:14.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:14.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:14.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb30 con 0x7f834c0f4690 2021-03-09T09:25:14.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c760 con 0x7f833801bab0 2021-03-09T09:25:14.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027830 con 0x7f8338062ca0 2021-03-09T09:25:14.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:25:14.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:14.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:25:14.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:14.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:14.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:14.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:14.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:14.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:14.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.689+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e680 con 0x7f833801bab0 2021-03-09T09:25:14.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.693+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 rx=0x7f8334028430 tx=0x7f833404e1e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:14.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.693+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833405fd70 con 0x7f833801bab0 2021-03-09T09:25:14.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.693+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833405f960 con 0x7f833801bab0 2021-03-09T09:25:14.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:14.697+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340248d0 con 0x7f833801bab0 2021-03-09T09:25:15.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:25:14 smithi146 bash[13136]: debug 2021-03-09T09:25:14.691+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6752 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:25:19.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:25:19.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:19.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 rx=0x7f8334028430 tx=0x7f833404e1e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:19.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 rx=0x7f8334028430 tx=0x7f833404e1e0).stop 2021-03-09T09:25:19.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:19.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:19.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:19.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:19.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:19.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380284b0 con 0x7f834c0f4690 2021-03-09T09:25:19.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027830 con 0x7f8338026c80 2021-03-09T09:25:19.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062520 con 0x7f8338062ca0 2021-03-09T09:25:19.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:25:19.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:19.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:25:19.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:19.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.689+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:19.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.693+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:19.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.693+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:19.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.693+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:19.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.693+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:19.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.693+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802c820 con 0x7f8338026c80 2021-03-09T09:25:19.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.693+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:19.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.693+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f8338026c80 2021-03-09T09:25:19.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.693+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f8338026c80 2021-03-09T09:25:19.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:19.697+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f8338026c80 2021-03-09T09:25:20.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:25:19 smithi146 bash[13136]: debug 2021-03-09T09:25:19.691+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6776 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:25:24.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.689+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:25:24.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.689+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:24.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.689+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:24.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.689+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c04e870).stop 2021-03-09T09:25:24.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.693+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338063080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:24.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.693+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338063080 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:24.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.693+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f83380162d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:24.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.693+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:24.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.693+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:24.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.693+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eea0 con 0x7f834c0f4690 2021-03-09T09:25:24.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.693+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:25:24.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.693+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:24.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.693+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f83380162d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:24.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.693+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:25:24.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.693+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:24.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.693+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb30 con 0x7f833801bab0 2021-03-09T09:25:24.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.693+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027830 con 0x7f8338062ca0 2021-03-09T09:25:24.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.697+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:24.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.697+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:24.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.697+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:24.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.697+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:24.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.697+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a710 con 0x7f833801bab0 2021-03-09T09:25:24.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.697+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f83380162d0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 rx=0x7f8334045920 tx=0x7f833404e1e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:24.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.697+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f833801bab0 2021-03-09T09:25:24.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.697+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334047cf0 con 0x7f833801bab0 2021-03-09T09:25:24.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:24.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403ade0 con 0x7f833801bab0 2021-03-09T09:25:25.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:25:24 smithi146 bash[13136]: debug 2021-03-09T09:25:24.690+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6802 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:25:29.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f83380162d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:25:29.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f83380162d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:29.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f83380162d0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 rx=0x7f8334045920 tx=0x7f833404e1e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:29.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f83380162d0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 rx=0x7f8334045920 tx=0x7f833404e1e0).stop 2021-03-09T09:25:29.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f83380162d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:29.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f83380162d0 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:29.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:29.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338061aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:29.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:29.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380284b0 con 0x7f834c0f4690 2021-03-09T09:25:29.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380695a0 con 0x7f8338026c80 2021-03-09T09:25:29.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027830 con 0x7f8338062ca0 2021-03-09T09:25:29.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:25:29.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:29.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:25:29.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:29.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.693+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338061aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:29.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.697+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:29.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.697+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:29.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.697+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:29.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.697+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:29.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.697+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064290 con 0x7f8338026c80 2021-03-09T09:25:29.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.697+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338061aa0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 rx=0x7f83440045c0 tx=0x7f834401f250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:29.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.697+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029030 con 0x7f8338026c80 2021-03-09T09:25:29.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.697+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f8338026c80 2021-03-09T09:25:29.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:29.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404ad20 con 0x7f8338026c80 2021-03-09T09:25:30.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:25:29 smithi146 bash[13136]: debug 2021-03-09T09:25:29.694+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6826 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:25:34.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338061aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:25:34.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338061aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:34.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338061aa0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 rx=0x7f83440045c0 tx=0x7f834401f250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:34.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338061aa0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 rx=0x7f83440045c0 tx=0x7f834401f250).stop 2021-03-09T09:25:34.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338061aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:34.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338061aa0 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:34.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:34.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:34.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:34.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:25:34.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:34.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:25:34.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:34.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a3d0 con 0x7f834c0f4690 2021-03-09T09:25:34.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013be0 con 0x7f833801bab0 2021-03-09T09:25:34.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380279e0 con 0x7f8338062ca0 2021-03-09T09:25:34.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:34.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:34.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:34.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:34.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:34.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380152a0 con 0x7f833801bab0 2021-03-09T09:25:34.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.697+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 rx=0x7f83440774e0 tx=0x7f834401f250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:34.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834407a120 con 0x7f833801bab0 2021-03-09T09:25:34.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404ad20 con 0x7f833801bab0 2021-03-09T09:25:34.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:34.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404b5f0 con 0x7f833801bab0 2021-03-09T09:25:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:25:34 smithi146 bash[13136]: debug 2021-03-09T09:25:34.694+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6852 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:25:39.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:25:39.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:39.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 rx=0x7f83440774e0 tx=0x7f834401f250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:39.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 rx=0x7f83440774e0 tx=0x7f834401f250).stop 2021-03-09T09:25:39.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:39.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:39.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:39.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:39.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:39.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:25:39.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:25:39.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:39.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:39.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb30 con 0x7f834c0f4690 2021-03-09T09:25:39.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027830 con 0x7f8338026c80 2021-03-09T09:25:39.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009db0 con 0x7f8338062ca0 2021-03-09T09:25:39.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:39.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:39.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:39.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:39.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:39.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064290 con 0x7f8338026c80 2021-03-09T09:25:39.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.697+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806fca0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:39.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.705+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f8338026c80 2021-03-09T09:25:39.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.705+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f8338026c80 2021-03-09T09:25:39.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:39.705+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0227a0 con 0x7f8338026c80 2021-03-09T09:25:40.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:25:39 smithi146 bash[13136]: debug 2021-03-09T09:25:39.698+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6876 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:25:44.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:25:44.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:44.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806fca0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:44.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806fca0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c04e870).stop 2021-03-09T09:25:44.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833806fca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:44.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806fca0 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:44.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f8338063790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:44.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:44.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:44.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006040 con 0x7f834c0f4690 2021-03-09T09:25:44.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bbc0 con 0x7f833801bab0 2021-03-09T09:25:44.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb30 con 0x7f8338062ca0 2021-03-09T09:25:44.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:25:44.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:44.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:25:44.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:44.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f8338063790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:44.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:44.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:44.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:44.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:44.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338019f30 con 0x7f833801bab0 2021-03-09T09:25:44.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f8338063790 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 rx=0x7f833400d010 tx=0x7f833404e1e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:44.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032080 con 0x7f833801bab0 2021-03-09T09:25:44.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334052450 con 0x7f833801bab0 2021-03-09T09:25:44.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:44.705+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334024130 con 0x7f833801bab0 2021-03-09T09:25:45.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:25:44 smithi146 bash[13136]: debug 2021-03-09T09:25:44.698+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6901 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:25:49.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f8338063790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:25:49.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f8338063790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:49.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f8338063790 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 rx=0x7f833400d010 tx=0x7f833404e1e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:49.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f8338063790 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 rx=0x7f833400d010 tx=0x7f833404e1e0).stop 2021-03-09T09:25:49.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f8338063790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:49.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f8338063790 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:49.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:49.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:49.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:49.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:25:49.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:49.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009db0 con 0x7f834c0f4690 2021-03-09T09:25:49.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ede0 con 0x7f8338026c80 2021-03-09T09:25:49.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801fda0 con 0x7f8338062ca0 2021-03-09T09:25:49.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:25:49.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:49.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:49.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:49.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:49.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:49.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:49.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a820 con 0x7f8338026c80 2021-03-09T09:25:49.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.701+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 rx=0x7f833c009540 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:49.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.705+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f8338026c80 2021-03-09T09:25:49.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.705+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0042e0 con 0x7f8338026c80 2021-03-09T09:25:49.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:49.709+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338026c80 2021-03-09T09:25:50.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:25:49 smithi146 bash[13136]: debug 2021-03-09T09:25:49.698+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6926 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:25:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:25:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 rx=0x7f833c009540 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 rx=0x7f833c009540 tx=0x7f833c04e870).stop 2021-03-09T09:25:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800d1a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027830 con 0x7f834c0f4690 2021-03-09T09:25:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb30 con 0x7f833801bab0 2021-03-09T09:25:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e530 con 0x7f8338062ca0 2021-03-09T09:25:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:25:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:25:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.701+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.705+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.705+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.705+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:54.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.705+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:54.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.705+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a710 con 0x7f833801bab0 2021-03-09T09:25:54.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.705+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 rx=0x7f8334064c10 tx=0x7f8334007c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:54.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.705+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032040 con 0x7f833801bab0 2021-03-09T09:25:54.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.705+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404ded0 con 0x7f833801bab0 2021-03-09T09:25:54.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:54.709+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402da60 con 0x7f833801bab0 2021-03-09T09:25:55.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:25:54 smithi146 bash[13136]: debug 2021-03-09T09:25:54.702+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6951 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:25:59.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:25:59.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:59.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 rx=0x7f8334064c10 tx=0x7f8334007c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:59.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 rx=0x7f8334064c10 tx=0x7f8334007c30).stop 2021-03-09T09:25:59.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:59.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:59.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:59.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:59.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:59.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021960 con 0x7f834c0f4690 2021-03-09T09:25:59.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005880 con 0x7f8338026c80 2021-03-09T09:25:59.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f8338062ca0 2021-03-09T09:25:59.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:25:59.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:59.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:25:59.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:59.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.704+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:59.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.708+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:59.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.708+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:59.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.708+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:59.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.708+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:59.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.708+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800be80 con 0x7f8338026c80 2021-03-09T09:25:59.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.708+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f83440724a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:59.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f8338026c80 2021-03-09T09:25:59.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072190 con 0x7f8338026c80 2021-03-09T09:25:59.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:25:59.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834407a120 con 0x7f8338026c80 2021-03-09T09:26:00.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:25:59 smithi146 bash[13136]: debug 2021-03-09T09:25:59.702+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 6976 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:26:04.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:26:04.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:04.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f83440724a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f83440724a0).stop 2021-03-09T09:26:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800d1a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:04.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:04.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:04.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c760 con 0x7f834c0f4690 2021-03-09T09:26:04.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027830 con 0x7f833801bab0 2021-03-09T09:26:04.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009410 con 0x7f8338062ca0 2021-03-09T09:26:04.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:26:04.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:04.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:26:04.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:04.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.704+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:04.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.708+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:04.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.708+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:04.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.708+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:04.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.708+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:04.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.708+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801ed90 con 0x7f833801bab0 2021-03-09T09:26:04.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.708+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 rx=0x7f8334009b40 tx=0x7f8334007c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:04.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.708+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f833801bab0 2021-03-09T09:26:04.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.708+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833406d360 con 0x7f833801bab0 2021-03-09T09:26:04.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:04.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f833801bab0 2021-03-09T09:26:05.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:26:04 smithi146 bash[13136]: debug 2021-03-09T09:26:04.706+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7001 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:26:09.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.704+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:26:09.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.704+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:09.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.704+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 rx=0x7f8334009b40 tx=0x7f8334007c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:09.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.704+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 rx=0x7f8334009b40 tx=0x7f8334007c30).stop 2021-03-09T09:26:09.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.704+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:09.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.704+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.704+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.704+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.704+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.704+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023320 con 0x7f834c0f4690 2021-03-09T09:26:09.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.704+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f8338026c80 2021-03-09T09:26:09.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.704+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017b70 con 0x7f8338062ca0 2021-03-09T09:26:09.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.704+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:26:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.704+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.708+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:26:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.708+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.708+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:09.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.708+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.712+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.712+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.712+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.712+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a710 con 0x7f8338026c80 2021-03-09T09:26:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.712+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 rx=0x7f834406a0b0 tx=0x7f8344004f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029040 con 0x7f8338026c80 2021-03-09T09:26:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344079e90 con 0x7f8338026c80 2021-03-09T09:26:09.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:09.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834407a120 con 0x7f8338026c80 2021-03-09T09:26:10.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:26:09 smithi146 bash[13136]: debug 2021-03-09T09:26:09.706+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7024 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:26:14.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:26:14.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:14.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 rx=0x7f834406a0b0 tx=0x7f8344004f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:14.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 rx=0x7f834406a0b0 tx=0x7f8344004f70).stop 2021-03-09T09:26:14.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800d1a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:14.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800d1a0 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:14.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:14.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:14.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:14.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f330 con 0x7f834c0f4690 2021-03-09T09:26:14.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021960 con 0x7f833801bab0 2021-03-09T09:26:14.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb30 con 0x7f8338062ca0 2021-03-09T09:26:14.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:26:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:26:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:14.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:14.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:14.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:14.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:14.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338070720 con 0x7f833801bab0 2021-03-09T09:26:14.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.708+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:14.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f833801bab0 2021-03-09T09:26:14.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833801bab0 2021-03-09T09:26:14.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:14.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c053270 con 0x7f833801bab0 2021-03-09T09:26:15.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:26:14 smithi146 bash[13136]: debug 2021-03-09T09:26:14.710+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7051 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:26:19.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:26:19.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:19.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:19.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c04e870).stop 2021-03-09T09:26:19.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:19.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:19.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:19.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:19.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338011970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:19.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027830 con 0x7f834c0f4690 2021-03-09T09:26:19.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016b50 con 0x7f8338026c80 2021-03-09T09:26:19.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064cd0 con 0x7f8338062ca0 2021-03-09T09:26:19.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:26:19.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:19.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338011970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:19.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:26:19.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:19.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:19.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:19.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:19.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:19.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017b70 con 0x7f8338026c80 2021-03-09T09:26:19.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338011970 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 rx=0x7f83380692d0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:19.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0042b0 con 0x7f8338026c80 2021-03-09T09:26:19.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c022730 con 0x7f8338026c80 2021-03-09T09:26:19.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:19.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c020cc0 con 0x7f8338026c80 2021-03-09T09:26:20.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:26:19 smithi146 bash[13136]: debug 2021-03-09T09:26:19.710+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7076 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:26:24.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338011970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:26:24.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338011970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:24.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338011970 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 rx=0x7f83380692d0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:24.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338011970 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 rx=0x7f83380692d0 tx=0x7f833c04e870).stop 2021-03-09T09:26:24.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338011970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:24.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338011970 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:24.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:24.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833800ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:24.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:24.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023320 con 0x7f834c0f4690 2021-03-09T09:26:24.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f833801bab0 2021-03-09T09:26:24.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027830 con 0x7f8338062ca0 2021-03-09T09:26:24.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:26:24.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:24.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:26:24.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:24.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833800ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:24.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833802d280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:24.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833802d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:24.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:24.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:24.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338019f00 con 0x7f833801bab0 2021-03-09T09:26:24.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.712+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833800ddc0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344029000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:24.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f833801bab0 2021-03-09T09:26:24.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072190 con 0x7f833801bab0 2021-03-09T09:26:24.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:24.720+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833801bab0 2021-03-09T09:26:25.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:26:24 smithi146 bash[13136]: debug 2021-03-09T09:26:24.713+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7101 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:26:29.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833800ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:26:29.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833800ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:29.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833800ddc0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344029000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:29.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833800ddc0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344029000).stop 2021-03-09T09:26:29.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833800ddc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:29.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833800ddc0 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:29.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:29.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338017d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:29.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338018280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:29.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005810 con 0x7f834c0f4690 2021-03-09T09:26:29.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027830 con 0x7f8338026c80 2021-03-09T09:26:29.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016b50 con 0x7f8338062ca0 2021-03-09T09:26:29.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338018280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:26:29.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338018280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:29.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:26:29.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:29.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338017d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:29.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338018280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:29.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338018280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:29.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:29.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:29.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800be80 con 0x7f8338026c80 2021-03-09T09:26:29.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338017d90 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 rx=0x7f8344016930 tx=0x7f8344029000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:29.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344016770 con 0x7f8338026c80 2021-03-09T09:26:29.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344016590 con 0x7f8338026c80 2021-03-09T09:26:29.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:29.720+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344071130 con 0x7f8338026c80 2021-03-09T09:26:30.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:26:29 smithi146 bash[13136]: debug 2021-03-09T09:26:29.713+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7126 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:26:34.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338017d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:26:34.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338017d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:34.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338017d90 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 rx=0x7f8344016930 tx=0x7f8344029000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:34.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338017d90 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 rx=0x7f8344016930 tx=0x7f8344029000).stop 2021-03-09T09:26:34.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338017d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:34.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338017d90 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:34.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:34.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:34.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:34.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f834c0f4690 2021-03-09T09:26:34.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f833801bab0 2021-03-09T09:26:34.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023320 con 0x7f8338062ca0 2021-03-09T09:26:34.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833801e390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:26:34.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:34.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:26:34.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:34.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:34.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833801e390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:34.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:34.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:34.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:34.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338070720 con 0x7f833801bab0 2021-03-09T09:26:34.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.716+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 rx=0x7f8334070af0 tx=0x7f8334007c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:34.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.720+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f833801bab0 2021-03-09T09:26:34.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.720+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340079c0 con 0x7f833801bab0 2021-03-09T09:26:34.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:34.724+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d360 con 0x7f833801bab0 2021-03-09T09:26:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:26:34 smithi146 bash[13136]: debug 2021-03-09T09:26:34.713+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7150 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:26:39.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:26:39.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:39.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 rx=0x7f8334070af0 tx=0x7f8334007c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:39.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 rx=0x7f8334070af0 tx=0x7f8334007c30).stop 2021-03-09T09:26:39.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:39.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:39.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:39.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:39.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:39.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005810 con 0x7f834c0f4690 2021-03-09T09:26:39.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380109b0 con 0x7f8338026c80 2021-03-09T09:26:39.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f8338062ca0 2021-03-09T09:26:39.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:26:39.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:39.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833801e390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:26:39.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:39.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:39.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833801e390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:39.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:39.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:39.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:39.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800fa40 con 0x7f8338026c80 2021-03-09T09:26:39.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800a420 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 rx=0x7f833404b4e0 tx=0x7f8334007ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:39.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f8338026c80 2021-03-09T09:26:39.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.720+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403ae20 con 0x7f8338026c80 2021-03-09T09:26:39.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:39.724+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334027990 con 0x7f8338026c80 2021-03-09T09:26:40.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:26:39 smithi146 bash[13136]: debug 2021-03-09T09:26:39.717+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7176 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:26:44.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:26:44.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:44.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800a420 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 rx=0x7f833404b4e0 tx=0x7f8334007ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:44.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800a420 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 rx=0x7f833404b4e0 tx=0x7f8334007ca0).stop 2021-03-09T09:26:44.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800a420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:44.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800a420 unknown :-1 s=CLOSED pgs=2590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:44.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:44.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:44.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:44.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023320 con 0x7f834c0f4690 2021-03-09T09:26:44.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027830 con 0x7f833801bab0 2021-03-09T09:26:44.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005810 con 0x7f8338062ca0 2021-03-09T09:26:44.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833801e390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:26:44.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:44.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:26:44.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:44.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.716+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:44.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.720+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833801e390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:44.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.720+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:44.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.720+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:44.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.720+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:44.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.720+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800b470 con 0x7f833801bab0 2021-03-09T09:26:44.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.720+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 rx=0x7f833406d8f0 tx=0x7f8334007ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:44.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.720+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f833801bab0 2021-03-09T09:26:44.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.720+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334026630 con 0x7f833801bab0 2021-03-09T09:26:44.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:44.724+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403b290 con 0x7f833801bab0 2021-03-09T09:26:45.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:26:44 smithi146 bash[13136]: debug 2021-03-09T09:26:44.717+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7201 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:26:49.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:26:49.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:49.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 rx=0x7f833406d8f0 tx=0x7f8334007ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:49.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 rx=0x7f833406d8f0 tx=0x7f8334007ca0).stop 2021-03-09T09:26:49.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:49.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:49.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:49.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:49.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:49.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061550 con 0x7f834c0f4690 2021-03-09T09:26:49.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f8338026c80 2021-03-09T09:26:49.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005810 con 0x7f8338062ca0 2021-03-09T09:26:49.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:26:49.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:49.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833801e390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:26:49.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:49.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:49.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833801e390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:49.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:49.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:49.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:49.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a710 con 0x7f8338026c80 2021-03-09T09:26:49.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.719+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802c9e0 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 rx=0x7f8334007e00 tx=0x7f8334007ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:49.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.723+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f8338026c80 2021-03-09T09:26:49.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.723+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340603a0 con 0x7f8338026c80 2021-03-09T09:26:49.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:49.723+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334033280 con 0x7f8338026c80 2021-03-09T09:26:50.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:26:49 smithi146 bash[13136]: debug 2021-03-09T09:26:49.721+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7226 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:26:54.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833802c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:26:54.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833802c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:54.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802c9e0 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 rx=0x7f8334007e00 tx=0x7f8334007ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:54.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802c9e0 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 rx=0x7f8334007e00 tx=0x7f8334007ca0).stop 2021-03-09T09:26:54.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833802c9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:54.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802c9e0 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:54.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338024400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:54.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021b60 con 0x7f834c0f4690 2021-03-09T09:26:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380123f0 con 0x7f833801bab0 2021-03-09T09:26:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a870 con 0x7f8338062ca0 2021-03-09T09:26:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338024400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:26:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338024400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:26:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.723+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:54.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.727+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338024400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:54.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.727+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338024400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.727+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.727+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.727+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806fae0 con 0x7f833801bab0 2021-03-09T09:26:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.727+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 rx=0x7f833c019e70 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:54.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c006ea0 con 0x7f833801bab0 2021-03-09T09:26:54.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f833801bab0 2021-03-09T09:26:54.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:54.731+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c069270 con 0x7f833801bab0 2021-03-09T09:26:55.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:26:54 smithi146 bash[13136]: debug 2021-03-09T09:26:54.721+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7247 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:26:59.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.723+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:26:59.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.723+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:59.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.723+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 rx=0x7f833c019e70 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:59.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.723+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 rx=0x7f833c019e70 tx=0x7f833c04e870).stop 2021-03-09T09:26:59.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.723+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:59.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.723+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:59.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.723+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:59.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.727+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:59.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.727+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:59.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380109b0 con 0x7f834c0f4690 2021-03-09T09:26:59.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027830 con 0x7f8338026c80 2021-03-09T09:26:59.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005810 con 0x7f8338062ca0 2021-03-09T09:26:59.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.727+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:26:59.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.727+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:59.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.727+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:59.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.727+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800fe70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:26:59.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.727+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800fe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:59.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.727+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800fe70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:59.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.727+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800fe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:59.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.727+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:59.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.727+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:59.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.727+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338008800 con 0x7f8338026c80 2021-03-09T09:26:59.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.731+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f980 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 rx=0x7f8334060560 tx=0x7f8334007ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:59.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.731+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334032070 con 0x7f8338026c80 2021-03-09T09:26:59.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.735+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334070af0 con 0x7f8338026c80 2021-03-09T09:26:59.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:26:59.735+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334026de0 con 0x7f8338026c80 2021-03-09T09:27:00.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:26:59 smithi146 bash[13136]: debug 2021-03-09T09:26:59.721+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7275 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:27:04.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:27:04.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:04.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f980 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 rx=0x7f8334060560 tx=0x7f8334007ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:04.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f980 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 rx=0x7f8334060560 tx=0x7f8334007ca0).stop 2021-03-09T09:27:04.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800f980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:04.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800f980 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d450 con 0x7f834c0f4690 2021-03-09T09:27:04.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021b60 con 0x7f833801bab0 2021-03-09T09:27:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005810 con 0x7f8338062ca0 2021-03-09T09:27:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:27:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800fe70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:27:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800fe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.727+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.731+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800fe70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.731+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800fe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.731+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.731+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.731+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338070720 con 0x7f833801bab0 2021-03-09T09:27:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.731+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344029000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:04.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.731+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344072190 con 0x7f833801bab0 2021-03-09T09:27:04.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.735+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402b760 con 0x7f833801bab0 2021-03-09T09:27:04.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:04.735+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344028d20 con 0x7f833801bab0 2021-03-09T09:27:05.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:27:04 smithi146 bash[13136]: debug 2021-03-09T09:27:04.721+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7301 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:27:09.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.727+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:27:09.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.727+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:09.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.727+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344029000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:09.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.727+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344029000).stop 2021-03-09T09:27:09.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:09.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.727+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:09.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.727+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338067090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:09.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.727+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:09.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.727+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:09.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833805eb50 con 0x7f834c0f4690 2021-03-09T09:27:09.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380109b0 con 0x7f8338026c80 2021-03-09T09:27:09.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005810 con 0x7f8338062ca0 2021-03-09T09:27:09.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.727+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:27:09.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.727+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:09.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.727+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800fe70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:27:09.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.727+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800fe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:09.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.731+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338067090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:09.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.731+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800fe70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:09.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.731+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800fe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:09.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.731+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:09.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.731+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:09.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.731+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800b470 con 0x7f8338026c80 2021-03-09T09:27:09.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.731+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338067090 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334007ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:09.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.731+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f8338026c80 2021-03-09T09:27:09.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.731+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334060360 con 0x7f8338026c80 2021-03-09T09:27:09.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:09.735+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401cb50 con 0x7f8338026c80 2021-03-09T09:27:10.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:27:09 smithi146 bash[13136]: debug 2021-03-09T09:27:09.725+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7325 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:27:14.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.727+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338067090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:27:14.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.727+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338067090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.727+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338067090 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334007ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.727+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338067090 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334007ca0).stop 2021-03-09T09:27:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338067090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.727+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338067090 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:14.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.727+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:14.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.727+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:14.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.727+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:14.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.727+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:27:14.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380109b0 con 0x7f834c0f4690 2021-03-09T09:27:14.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.727+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:14.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ab20 con 0x7f833801bab0 2021-03-09T09:27:14.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021b60 con 0x7f8338062ca0 2021-03-09T09:27:14.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.727+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833801e390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:27:14.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.731+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:14.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.731+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:14.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.731+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833801e390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:14.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.731+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:14.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.731+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:14.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.731+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:14.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.731+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380288b0 con 0x7f833801bab0 2021-03-09T09:27:14.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.731+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 rx=0x7f834404acb0 tx=0x7f83440134d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:14.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.735+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029040 con 0x7f833801bab0 2021-03-09T09:27:14.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.735+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833801bab0 2021-03-09T09:27:14.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:14.739+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833801bab0 2021-03-09T09:27:15.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:27:14 smithi146 bash[13136]: debug 2021-03-09T09:27:14.728+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7351 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:27:19.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:27:19.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:19.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 rx=0x7f834404acb0 tx=0x7f83440134d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:19.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 rx=0x7f834404acb0 tx=0x7f83440134d0).stop 2021-03-09T09:27:19.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:19.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:19.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:19.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338014980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:19.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:19.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380109b0 con 0x7f834c0f4690 2021-03-09T09:27:19.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006930 con 0x7f8338026c80 2021-03-09T09:27:19.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833805eb50 con 0x7f8338062ca0 2021-03-09T09:27:19.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833801e390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:27:19.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:27:19.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:19.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:19.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.731+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338014980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:19.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.735+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833801e390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:19.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.735+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833801e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:19.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.735+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:19.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.735+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:19.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.735+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800b470 con 0x7f8338026c80 2021-03-09T09:27:19.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.735+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338014980 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f83440134d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:19.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.735+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029030 con 0x7f8338026c80 2021-03-09T09:27:19.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.735+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834407ad60 con 0x7f8338026c80 2021-03-09T09:27:19.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:19.739+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404b6d0 con 0x7f8338026c80 2021-03-09T09:27:20.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:27:19 smithi146 bash[13136]: debug 2021-03-09T09:27:19.728+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7375 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:27:24.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.731+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338014980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:27:24.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.731+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338014980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:24.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.731+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338014980 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f83440134d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:24.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.731+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338014980 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f83440134d0).stop 2021-03-09T09:27:24.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.731+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338014980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:24.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.731+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338014980 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:24.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.731+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:24.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.731+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:24.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.731+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:24.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.731+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833805eb50 con 0x7f834c0f4690 2021-03-09T09:27:24.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.731+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006930 con 0x7f833801bab0 2021-03-09T09:27:24.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.731+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062730 con 0x7f8338062ca0 2021-03-09T09:27:24.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.731+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:27:24.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.731+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:24.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.735+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:24.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.735+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:27:24.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.735+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:24.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.735+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:24.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.735+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:24.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.735+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:24.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.735+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:24.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.735+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380123a0 con 0x7f833801bab0 2021-03-09T09:27:24.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.735+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f83440134d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:24.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.739+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029040 con 0x7f833801bab0 2021-03-09T09:27:24.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.739+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440774b0 con 0x7f833801bab0 2021-03-09T09:27:24.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:24.743+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833801bab0 2021-03-09T09:27:25.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:27:24 smithi146 bash[13136]: debug 2021-03-09T09:27:24.732+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7401 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:27:29.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:27:29.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:29.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f83440134d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:29.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f83440134d0).stop 2021-03-09T09:27:29.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:29.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:29.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:29.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:29.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:29.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380123f0 con 0x7f834c0f4690 2021-03-09T09:27:29.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b7b0 con 0x7f8338026c80 2021-03-09T09:27:29.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015ad0 con 0x7f8338062ca0 2021-03-09T09:27:29.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:27:29.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:29.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:27:29.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:29.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:29.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:29.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:29.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:29.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:29.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800b470 con 0x7f8338026c80 2021-03-09T09:27:29.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.735+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800fe70 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 rx=0x7f8334052500 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:29.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.739+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338026c80 2021-03-09T09:27:29.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.739+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000d20 con 0x7f8338026c80 2021-03-09T09:27:29.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:29.743+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334031320 con 0x7f8338026c80 2021-03-09T09:27:30.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:27:29 smithi146 bash[13136]: debug 2021-03-09T09:27:29.732+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7424 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:27:34.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:27:34.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:34.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800fe70 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 rx=0x7f8334052500 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:34.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800fe70 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 rx=0x7f8334052500 tx=0x7f8334069000).stop 2021-03-09T09:27:34.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800fe70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:34.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800fe70 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:34.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:34.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:34.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:34.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f834c0f4690 2021-03-09T09:27:34.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380067a0 con 0x7f833801bab0 2021-03-09T09:27:34.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bbb0 con 0x7f8338062ca0 2021-03-09T09:27:34.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:27:34.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:34.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:27:34.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:34.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.734+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:34.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.738+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:34.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.738+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:34.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.738+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:34.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.738+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:34.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.738+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380068e0 con 0x7f833801bab0 2021-03-09T09:27:34.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.738+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f83440134d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:34.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f833801bab0 2021-03-09T09:27:34.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440770f0 con 0x7f833801bab0 2021-03-09T09:27:34.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:34.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400d450 con 0x7f833801bab0 2021-03-09T09:27:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:27:34 smithi146 bash[13136]: debug 2021-03-09T09:27:34.732+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7451 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:27:39.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.734+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:27:39.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.734+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:39.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.734+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f83440134d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:39.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.734+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f83440134d0).stop 2021-03-09T09:27:39.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:39.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.738+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:39.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.738+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833801b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:39.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.738+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:39.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.738+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:39.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006930 con 0x7f834c0f4690 2021-03-09T09:27:39.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380123f0 con 0x7f8338026c80 2021-03-09T09:27:39.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c860 con 0x7f8338062ca0 2021-03-09T09:27:39.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.738+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:27:39.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.738+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:39.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.738+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:27:39.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.738+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:39.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.738+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833801b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:39.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.746+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:39.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.746+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:39.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.746+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:39.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.746+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:39.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.746+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338067450 con 0x7f8338026c80 2021-03-09T09:27:39.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.746+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833801b470 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 rx=0x7f833402fd60 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:39.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334064220 con 0x7f8338026c80 2021-03-09T09:27:39.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334064220 con 0x7f8338026c80 2021-03-09T09:27:39.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:39.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334031b50 con 0x7f8338026c80 2021-03-09T09:27:40.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:27:39 smithi146 bash[13136]: debug 2021-03-09T09:27:39.736+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7474 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:27:44.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833801b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:27:44.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833801b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:44.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833801b470 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 rx=0x7f833402fd60 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:44.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833801b470 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 rx=0x7f833402fd60 tx=0x7f8334069000).stop 2021-03-09T09:27:44.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833801b470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:44.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833801b470 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:44.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:44.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:44.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:44.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f834c0f4690 2021-03-09T09:27:44.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380123f0 con 0x7f833801bab0 2021-03-09T09:27:44.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015b10 con 0x7f8338062ca0 2021-03-09T09:27:44.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:27:44.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:27:44.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:44.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:44.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.738+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:44.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.742+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:44.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.742+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:44.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.742+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:44.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.742+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:44.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.742+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e730 con 0x7f833801bab0 2021-03-09T09:27:44.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.742+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834407aed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:44.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344048dd0 con 0x7f833801bab0 2021-03-09T09:27:44.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072190 con 0x7f833801bab0 2021-03-09T09:27:44.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:44.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833801bab0 2021-03-09T09:27:45.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:27:44 smithi146 bash[13136]: debug 2021-03-09T09:27:44.736+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7501 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:27:49.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.738+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:27:49.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.738+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:49.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.738+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834407aed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:49.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.738+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834407aed0).stop 2021-03-09T09:27:49.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:49.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.738+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:49.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.738+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:49.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.738+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:49.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.738+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:49.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013bf0 con 0x7f834c0f4690 2021-03-09T09:27:49.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dc10 con 0x7f8338026c80 2021-03-09T09:27:49.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8338062ca0 2021-03-09T09:27:49.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.738+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:27:49.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.738+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:49.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.742+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:27:49.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.742+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:49.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.742+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:49.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.742+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:49.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.742+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:49.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.742+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:49.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.742+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:49.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.742+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380068e0 con 0x7f8338026c80 2021-03-09T09:27:49.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.746+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800a9e0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 rx=0x7f8334031fd0 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:49.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401bc20 con 0x7f8338026c80 2021-03-09T09:27:49.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401adb0 con 0x7f8338026c80 2021-03-09T09:27:49.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:49.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401adb0 con 0x7f8338026c80 2021-03-09T09:27:50.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:27:49 smithi146 bash[13136]: debug 2021-03-09T09:27:49.740+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7524 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:27:54.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:27:54.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:54.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800a9e0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 rx=0x7f8334031fd0 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:54.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800a9e0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 rx=0x7f8334031fd0 tx=0x7f8334069000).stop 2021-03-09T09:27:54.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800a9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:54.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800a9e0 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:54.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:54.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:54.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:54.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dc10 con 0x7f834c0f4690 2021-03-09T09:27:54.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380123f0 con 0x7f833801bab0 2021-03-09T09:27:54.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f380 con 0x7f8338062ca0 2021-03-09T09:27:54.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:27:54.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:27:54.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:54.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:54.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:54.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:54.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:54.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:54.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:54.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.742+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028f20 con 0x7f833801bab0 2021-03-09T09:27:54.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.746+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 rx=0x7f833c018cc0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:54.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018ac0 con 0x7f833801bab0 2021-03-09T09:27:54.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833801bab0 2021-03-09T09:27:54.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:54.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c053270 con 0x7f833801bab0 2021-03-09T09:27:55.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:27:54 smithi146 bash[13136]: debug 2021-03-09T09:27:54.740+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7551 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:27:59.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.742+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:27:59.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.742+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:59.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.742+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 rx=0x7f833c018cc0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:59.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.742+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 rx=0x7f833c018cc0 tx=0x7f833c04e870).stop 2021-03-09T09:27:59.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:59.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.742+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.742+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380251c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.742+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.742+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005810 con 0x7f834c0f4690 2021-03-09T09:27:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dc10 con 0x7f8338026c80 2021-03-09T09:27:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f8338062ca0 2021-03-09T09:27:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.742+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:27:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.742+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:27:59.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.742+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:59.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.742+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380251c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:59.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.746+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f780 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-03-09T09:27:59.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.746+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:59.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.746+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:59.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.746+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:59.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.746+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:59.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.746+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e4d0 con 0x7f8338026c80 2021-03-09T09:27:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.746+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380251c0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f8338026c80 2021-03-09T09:27:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404d7b0 con 0x7f8338026c80 2021-03-09T09:27:59.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:27:59.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340339b0 con 0x7f8338026c80 2021-03-09T09:28:00.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:27:59 smithi146 bash[13136]: debug 2021-03-09T09:27:59.744+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7574 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:28:04.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f83380251c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:28:04.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f83380251c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:04.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380251c0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:04.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380251c0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f8334069000).stop 2021-03-09T09:28:04.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f83380251c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:04.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f83380251c0 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:04.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f380 con 0x7f834c0f4690 2021-03-09T09:28:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dc10 con 0x7f833801bab0 2021-03-09T09:28:04.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023e20 con 0x7f8338062ca0 2021-03-09T09:28:04.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:28:04.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:04.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:28:04.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:04.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:04.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.746+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380209e0 con 0x7f833801bab0 2021-03-09T09:28:04.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.750+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834407aed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:04.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f833801bab0 2021-03-09T09:28:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402b760 con 0x7f833801bab0 2021-03-09T09:28:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:04.754+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404ad90 con 0x7f833801bab0 2021-03-09T09:28:05.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:28:04 smithi146 bash[13136]: debug 2021-03-09T09:28:04.748+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7601 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:28:09.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:28:09.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:09.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834407aed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:09.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834407aed0).stop 2021-03-09T09:28:09.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:09.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:09.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:09.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:09.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:09.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f834c0f4690 2021-03-09T09:28:09.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f240 con 0x7f8338026c80 2021-03-09T09:28:09.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005810 con 0x7f8338062ca0 2021-03-09T09:28:09.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:28:09.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:28:09.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:09.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:09.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.746+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:09.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.750+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:09.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.750+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:09.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.750+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:09.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.750+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:09.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.750+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380087d0 con 0x7f8338026c80 2021-03-09T09:28:09.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.750+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834407aed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:09.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.754+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344054ea0 con 0x7f8338026c80 2021-03-09T09:28:09.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.754+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402b430 con 0x7f8338026c80 2021-03-09T09:28:09.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:09.754+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f8338026c80 2021-03-09T09:28:10.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:28:09 smithi146 bash[13136]: debug 2021-03-09T09:28:09.748+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7620 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:28:14.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.746+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:28:14.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.746+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:14.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.746+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834407aed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:14.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.746+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834407aed0).stop 2021-03-09T09:28:14.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:14.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:14.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:14.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:14.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:14.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dc10 con 0x7f834c0f4690 2021-03-09T09:28:14.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e5d0 con 0x7f833801bab0 2021-03-09T09:28:14.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8338062ca0 2021-03-09T09:28:14.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:28:14.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:14.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:14.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:28:14.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:14.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:14.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:14.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:14.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:14.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338061530 con 0x7f833801bab0 2021-03-09T09:28:14.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 rx=0x7f833c06a8a0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:14.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c06a4b0 con 0x7f833801bab0 2021-03-09T09:28:14.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.754+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c06a4b0 con 0x7f833801bab0 2021-03-09T09:28:14.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:14.754+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f833801bab0 2021-03-09T09:28:15.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:28:14 smithi146 bash[13136]: debug 2021-03-09T09:28:14.747+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7651 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:28:19.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:28:19.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:19.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 rx=0x7f833c06a8a0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:19.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 rx=0x7f833c06a8a0 tx=0x7f833c04e870).stop 2021-03-09T09:28:19.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:19.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:19.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:19.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:28:19.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:19.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338014960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:19.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:19.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380067a0 con 0x7f834c0f4690 2021-03-09T09:28:19.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a990 con 0x7f8338026c80 2021-03-09T09:28:19.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005810 con 0x7f8338062ca0 2021-03-09T09:28:19.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:28:19.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:19.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.750+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338014960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:19.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.754+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:19.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.754+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:19.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.754+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:19.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.754+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:19.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.754+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380068e0 con 0x7f8338026c80 2021-03-09T09:28:19.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.754+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338014960 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f834407aed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:19.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.754+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344010ef0 con 0x7f8338026c80 2021-03-09T09:28:19.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.754+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338026c80 2021-03-09T09:28:19.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:19.758+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400d450 con 0x7f8338026c80 2021-03-09T09:28:20.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:28:19 smithi146 bash[13136]: debug 2021-03-09T09:28:19.751+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7673 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:28:24.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338014960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:28:24.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338014960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:24.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338014960 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f834407aed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:24.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338014960 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f834407aed0).stop 2021-03-09T09:28:24.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f8338014960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:24.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f8338014960 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:24.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:24.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:28:24.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:24.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:24.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:24.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d920 con 0x7f834c0f4690 2021-03-09T09:28:24.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dc10 con 0x7f833801bab0 2021-03-09T09:28:24.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006930 con 0x7f8338062ca0 2021-03-09T09:28:24.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:28:24.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:24.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:24.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:24.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:24.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:24.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:24.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380665c0 con 0x7f833801bab0 2021-03-09T09:28:24.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834407aed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:24.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f833801bab0 2021-03-09T09:28:24.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072190 con 0x7f833801bab0 2021-03-09T09:28:24.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:24.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404ad20 con 0x7f833801bab0 2021-03-09T09:28:25.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:28:24 smithi146 bash[13136]: debug 2021-03-09T09:28:24.751+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7701 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:28:29.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:28:29.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:29.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834407aed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:29.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834407aed0).stop 2021-03-09T09:28:29.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:29.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801dea0 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:29.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:29.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:28:29.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:29.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:29.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:29.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013f90 con 0x7f834c0f4690 2021-03-09T09:28:29.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025a70 con 0x7f8338026c80 2021-03-09T09:28:29.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dc10 con 0x7f8338062ca0 2021-03-09T09:28:29.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:28:29.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:29.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.753+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:29.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.757+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:29.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.757+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:29.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.757+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:29.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.757+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:29.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.757+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800b470 con 0x7f8338026c80 2021-03-09T09:28:29.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.757+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 rx=0x7f8344004e50 tx=0x7f834407aed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:29.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.757+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400c0c0 con 0x7f8338026c80 2021-03-09T09:28:29.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.757+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400a040 con 0x7f8338026c80 2021-03-09T09:28:29.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:29.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404ad20 con 0x7f8338026c80 2021-03-09T09:28:30.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:28:29 smithi146 bash[13136]: debug 2021-03-09T09:28:29.755+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7724 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:28:34.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.753+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:28:34.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.753+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:34.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.753+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 rx=0x7f8344004e50 tx=0x7f834407aed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:34.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.753+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 rx=0x7f8344004e50 tx=0x7f834407aed0).stop 2021-03-09T09:28:34.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:34.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=CLOSED pgs=2779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:34.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:34.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:34.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:34.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068f30 con 0x7f834c0f4690 2021-03-09T09:28:34.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b330 con 0x7f833801bab0 2021-03-09T09:28:34.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8338062ca0 2021-03-09T09:28:34.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.753+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:28:34.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.753+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:34.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.753+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:28:34.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.753+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:34.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.757+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:34.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.757+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:34.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.757+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:34.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.757+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:34.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.757+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:34.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.757+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028410 con 0x7f833801bab0 2021-03-09T09:28:34.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.757+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801cbc0 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834407aed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:34.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344018d80 con 0x7f833801bab0 2021-03-09T09:28:34.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400a040 con 0x7f833801bab0 2021-03-09T09:28:34.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:34.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400d450 con 0x7f833801bab0 2021-03-09T09:28:35.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:28:34 smithi146 bash[13136]: debug 2021-03-09T09:28:34.755+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7751 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:28:39.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:28:39.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:39.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801cbc0 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834407aed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:39.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801cbc0 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834407aed0).stop 2021-03-09T09:28:39.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 msgr2=0x7f833801cbc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:39.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801bab0 0x7f833801cbc0 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:39.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:39.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:39.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:39.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025910 con 0x7f834c0f4690 2021-03-09T09:28:39.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f380 con 0x7f8338026c80 2021-03-09T09:28:39.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005810 con 0x7f8338062ca0 2021-03-09T09:28:39.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:28:39.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:39.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:39.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:28:39.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:39.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:39.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:39.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:39.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:39.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.757+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064400 con 0x7f8338026c80 2021-03-09T09:28:39.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.761+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:39.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04d1c0 con 0x7f8338026c80 2021-03-09T09:28:39.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f030 con 0x7f8338026c80 2021-03-09T09:28:39.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:39.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004110 con 0x7f8338026c80 2021-03-09T09:28:40.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:28:39 smithi146 bash[13136]: debug 2021-03-09T09:28:39.759+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7774 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:28:44.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:28:44.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:44.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:44.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04e870).stop 2021-03-09T09:28:44.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:44.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=CLOSED pgs=2796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:44.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:44.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:44.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833801d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:44.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013e50 con 0x7f834c0f4690 2021-03-09T09:28:44.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dc10 con 0x7f833801cbc0 2021-03-09T09:28:44.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005810 con 0x7f8338062ca0 2021-03-09T09:28:44.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:28:44.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:44.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:28:44.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833801d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:44.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:44.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:44.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:44.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:44.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:44.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338013f90 con 0x7f833801cbc0 2021-03-09T09:28:44.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833801d040 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 rx=0x7f833c06a470 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:44.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0606d0 con 0x7f833801cbc0 2021-03-09T09:28:44.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f833801cbc0 2021-03-09T09:28:44.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:44.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f833801cbc0 2021-03-09T09:28:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:28:44 smithi146 bash[13136]: debug 2021-03-09T09:28:44.759+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7800 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:28:49.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833801d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:28:49.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833801d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:49.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833801d040 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 rx=0x7f833c06a470 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:49.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833801d040 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 rx=0x7f833c06a470 tx=0x7f833c04e870).stop 2021-03-09T09:28:49.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833801d040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:49.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833801d040 unknown :-1 s=CLOSED pgs=2804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:49.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:49.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:49.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:28:49.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:49.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:49.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380183d0 con 0x7f834c0f4690 2021-03-09T09:28:49.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380050a0 con 0x7f8338026c80 2021-03-09T09:28:49.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380621d0 con 0x7f8338062ca0 2021-03-09T09:28:49.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:28:49.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:49.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.761+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:49.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:49.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:49.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:49.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:49.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c0a0 con 0x7f8338026c80 2021-03-09T09:28:49.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 rx=0x7f834400be60 tx=0x7f834407aed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:49.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344065000 con 0x7f8338026c80 2021-03-09T09:28:49.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.769+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338026c80 2021-03-09T09:28:49.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:49.769+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400d450 con 0x7f8338026c80 2021-03-09T09:28:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:28:49 smithi146 bash[13136]: debug 2021-03-09T09:28:49.759+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7824 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:28:54.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:28:54.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:54.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 rx=0x7f834400be60 tx=0x7f834407aed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:54.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 rx=0x7f834400be60 tx=0x7f834407aed0).stop 2021-03-09T09:28:54.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:54.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=CLOSED pgs=2818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:54.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:54.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:28:54.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:54.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833801d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:54.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:54.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025a70 con 0x7f834c0f4690 2021-03-09T09:28:54.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dc10 con 0x7f833801cbc0 2021-03-09T09:28:54.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066a30 con 0x7f8338062ca0 2021-03-09T09:28:54.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:28:54.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:54.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833801d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:54.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:54.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:54.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:54.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:54.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064400 con 0x7f833801cbc0 2021-03-09T09:28:54.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833801d040 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834407aed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:54.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833801cbc0 2021-03-09T09:28:54.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f833801cbc0 2021-03-09T09:28:54.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:54.769+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344072760 con 0x7f833801cbc0 2021-03-09T09:28:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:28:54 smithi146 bash[13136]: debug 2021-03-09T09:28:54.763+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7848 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:28:59.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833801d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:28:59.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833801d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:59.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833801d040 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834407aed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:59.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833801d040 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834407aed0).stop 2021-03-09T09:28:59.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833801d040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:59.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833801d040 unknown :-1 s=CLOSED pgs=2822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:59.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:59.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:59.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:59.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:28:59.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024210 con 0x7f834c0f4690 2021-03-09T09:28:59.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806fa80 con 0x7f8338026c80 2021-03-09T09:28:59.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:59.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8338062ca0 2021-03-09T09:28:59.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:28:59.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:59.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:59.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:59.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:59.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:59.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:59.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338018580 con 0x7f8338026c80 2021-03-09T09:28:59.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f834407aed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:59.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.769+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344072370 con 0x7f8338026c80 2021-03-09T09:28:59.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.769+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338026c80 2021-03-09T09:28:59.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:28:59.773+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440770f0 con 0x7f8338026c80 2021-03-09T09:29:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:28:59 smithi146 bash[13136]: debug 2021-03-09T09:28:59.763+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7874 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:29:04.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:29:04.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:04.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f834407aed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:04.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f834407aed0).stop 2021-03-09T09:29:04.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=CLOSED pgs=2832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338017720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:04.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066a30 con 0x7f834c0f4690 2021-03-09T09:29:04.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069da0 con 0x7f833801cbc0 2021-03-09T09:29:04.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380202c0 con 0x7f8338062ca0 2021-03-09T09:29:04.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:29:04.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:04.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338017720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:04.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:29:04.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:04.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.769+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:04.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.769+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:04.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.769+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:04.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.769+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:04.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.769+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c0a0 con 0x7f833801cbc0 2021-03-09T09:29:04.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.769+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338017720 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:04.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.769+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400e070 con 0x7f833801cbc0 2021-03-09T09:29:04.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.769+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403aeb0 con 0x7f833801cbc0 2021-03-09T09:29:04.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:04.773+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340077b0 con 0x7f833801cbc0 2021-03-09T09:29:05.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:29:04 smithi146 bash[13136]: debug 2021-03-09T09:29:04.767+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7900 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:29:09.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.764+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338017720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:29:09.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.764+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338017720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:09.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.764+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338017720 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:09.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.764+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338017720 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f8334069000).stop 2021-03-09T09:29:09.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.764+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338017720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:09.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.764+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338017720 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:09.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.764+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:09.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.764+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:09.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.764+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:09.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.764+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026580 con 0x7f834c0f4690 2021-03-09T09:29:09.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.764+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008160 con 0x7f8338026c80 2021-03-09T09:29:09.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.764+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015570 con 0x7f8338062ca0 2021-03-09T09:29:09.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.764+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:09.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.764+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:29:09.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.764+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:09.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.768+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:29:09.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.768+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:09.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.768+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:09.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.768+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:09.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.768+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:09.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.768+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:09.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.768+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f8338026c80 2021-03-09T09:29:09.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.768+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:09.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401ac10 con 0x7f8338026c80 2021-03-09T09:29:09.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340077b0 con 0x7f8338026c80 2021-03-09T09:29:09.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:09.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833405c8e0 con 0x7f8338026c80 2021-03-09T09:29:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:29:09 smithi146 bash[13136]: debug 2021-03-09T09:29:09.767+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7924 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:29:14.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:29:14.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:14.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:14.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).stop 2021-03-09T09:29:14.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:14.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=CLOSED pgs=2848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:14.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:14.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338015d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:14.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:14.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dc10 con 0x7f834c0f4690 2021-03-09T09:29:14.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025a70 con 0x7f833801cbc0 2021-03-09T09:29:14.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015570 con 0x7f8338062ca0 2021-03-09T09:29:14.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:29:14.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:29:14.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:14.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:14.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338015d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:14.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:14.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:14.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:14.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:14.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.768+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338018580 con 0x7f833801cbc0 2021-03-09T09:29:14.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.772+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338015d10 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344027250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:14.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f833801cbc0 2021-03-09T09:29:14.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440723a0 con 0x7f833801cbc0 2021-03-09T09:29:14.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:14.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834401f120 con 0x7f833801cbc0 2021-03-09T09:29:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:29:14 smithi146 bash[13136]: debug 2021-03-09T09:29:14.770+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7949 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:29:19.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338015d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:29:19.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338015d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:19.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338015d10 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344027250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:19.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338015d10 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344027250).stop 2021-03-09T09:29:19.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338015d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:19.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338015d10 unknown :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:19.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:19.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:19.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:19.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028410 con 0x7f834c0f4690 2021-03-09T09:29:19.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380106b0 con 0x7f8338026c80 2021-03-09T09:29:19.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028ff0 con 0x7f8338062ca0 2021-03-09T09:29:19.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:29:19.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:19.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:29:19.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:19.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:19.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:19.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:19.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:19.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:19.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c0a0 con 0x7f8338026c80 2021-03-09T09:29:19.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.772+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 rx=0x7f8338066e30 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:19.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c060980 con 0x7f8338026c80 2021-03-09T09:29:19.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8338026c80 2021-03-09T09:29:19.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:19.780+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338026c80 2021-03-09T09:29:20.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:29:19 smithi146 bash[13136]: debug 2021-03-09T09:29:19.770+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7974 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:29:24.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:29:24.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:24.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 rx=0x7f8338066e30 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:24.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 rx=0x7f8338066e30 tx=0x7f833c04e870).stop 2021-03-09T09:29:24.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:24.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=CLOSED pgs=2867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:24.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338062540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:24.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:24.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:24.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c5b0 con 0x7f834c0f4690 2021-03-09T09:29:24.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026580 con 0x7f833801cbc0 2021-03-09T09:29:24.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014d20 con 0x7f8338062ca0 2021-03-09T09:29:24.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:29:24.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:24.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338062540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:24.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:29:24.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:24.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:24.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:24.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:24.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:24.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068ee0 con 0x7f833801cbc0 2021-03-09T09:29:24.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.772+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338062540 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 rx=0x7f8334047d00 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:24.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f833801cbc0 2021-03-09T09:29:24.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402da60 con 0x7f833801cbc0 2021-03-09T09:29:24.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:24.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340645e0 con 0x7f833801cbc0 2021-03-09T09:29:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:29:24 smithi146 bash[13136]: debug 2021-03-09T09:29:24.774+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 7999 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:29:29.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.772+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338062540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:29:29.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.772+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338062540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:29.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.772+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338062540 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 rx=0x7f8334047d00 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:29.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.772+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338062540 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 rx=0x7f8334047d00 tx=0x7f8334069000).stop 2021-03-09T09:29:29.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338062540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:29.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.772+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338062540 unknown :-1 s=CLOSED pgs=2874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:29.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.772+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:29.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.772+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:29.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.772+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:29.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a990 con 0x7f834c0f4690 2021-03-09T09:29:29.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dc10 con 0x7f8338026c80 2021-03-09T09:29:29.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380181a0 con 0x7f8338062ca0 2021-03-09T09:29:29.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.772+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:29:29.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.772+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:29.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.772+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:29.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.776+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:29:29.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.776+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:29.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.776+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:29.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.776+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:29.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.776+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:29.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.776+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:29.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.776+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338013ba0 con 0x7f8338026c80 2021-03-09T09:29:29.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.776+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 rx=0x7f83380152c0 tx=0x7f8344027250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:29.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029030 con 0x7f8338026c80 2021-03-09T09:29:29.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344079550 con 0x7f8338026c80 2021-03-09T09:29:29.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:29.780+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440791e0 con 0x7f8338026c80 2021-03-09T09:29:30.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:29:29 smithi146 bash[13136]: debug 2021-03-09T09:29:29.774+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8024 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:29:34.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:29:34.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:34.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 rx=0x7f83380152c0 tx=0x7f8344027250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:34.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 rx=0x7f83380152c0 tx=0x7f8344027250).stop 2021-03-09T09:29:34.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:34.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:34.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833806ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:34.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:34.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:34.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071310 con 0x7f834c0f4690 2021-03-09T09:29:34.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026580 con 0x7f833801cbc0 2021-03-09T09:29:34.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014d20 con 0x7f8338062ca0 2021-03-09T09:29:34.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:29:34.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:29:34.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:34.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:34.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833806ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:34.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:34.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:34.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:34.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:34.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.776+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833805eb00 con 0x7f833801cbc0 2021-03-09T09:29:34.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.780+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833806ca90 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 rx=0x7f8334033380 tx=0x7f833405c520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:34.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.780+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334057cc0 con 0x7f833801cbc0 2021-03-09T09:29:34.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.780+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f833801cbc0 2021-03-09T09:29:34.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:34.784+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833406d180 con 0x7f833801cbc0 2021-03-09T09:29:35.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:29:34 smithi146 bash[13136]: debug 2021-03-09T09:29:34.778+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8049 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:29:39.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.776+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833806ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:29:39.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.776+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833806ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:39.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.776+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833806ca90 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 rx=0x7f8334033380 tx=0x7f833405c520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:39.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.776+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833806ca90 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 rx=0x7f8334033380 tx=0x7f833405c520).stop 2021-03-09T09:29:39.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833806ca90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:39.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.776+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833806ca90 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:39.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.776+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:39.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.776+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:39.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.776+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:39.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015570 con 0x7f834c0f4690 2021-03-09T09:29:39.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069fa0 con 0x7f8338026c80 2021-03-09T09:29:39.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025a70 con 0x7f8338062ca0 2021-03-09T09:29:39.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.776+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:29:39.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.776+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:39.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.776+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:29:39.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.776+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:39.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.780+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:39.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.788+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:39.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.788+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:39.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.788+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:39.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.788+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:39.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.788+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800eae0 con 0x7f8338026c80 2021-03-09T09:29:39.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.788+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 rx=0x7f8344054550 tx=0x7f8344027250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:39.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.788+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f8338026c80 2021-03-09T09:29:39.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.788+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344018eb0 con 0x7f8338026c80 2021-03-09T09:29:39.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:39.792+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400d490 con 0x7f8338026c80 2021-03-09T09:29:40.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:29:39 smithi146 bash[13136]: debug 2021-03-09T09:29:39.778+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8074 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:29:44.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:29:44.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:44.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 rx=0x7f8344054550 tx=0x7f8344027250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:44.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 rx=0x7f8344054550 tx=0x7f8344027250).stop 2021-03-09T09:29:44.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:44.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=CLOSED pgs=2904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:44.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338069320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:44.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:44.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:44.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380128e0 con 0x7f834c0f4690 2021-03-09T09:29:44.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071310 con 0x7f833801cbc0 2021-03-09T09:29:44.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014ce0 con 0x7f8338062ca0 2021-03-09T09:29:44.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:29:44.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:44.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:29:44.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:44.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338069320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:44.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f833800f660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:44.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f833800f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:44.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:44.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:44.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f833801cbc0 2021-03-09T09:29:44.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338069320 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 rx=0x7f8334009bd0 tx=0x7f833405c520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:44.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334000d50 con 0x7f833801cbc0 2021-03-09T09:29:44.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.784+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404d170 con 0x7f833801cbc0 2021-03-09T09:29:44.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:44.788+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404dc70 con 0x7f833801cbc0 2021-03-09T09:29:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:29:44 smithi146 bash[13136]: debug 2021-03-09T09:29:44.782+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8099 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:29:49.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T09:29:49.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:49.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338069320 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 rx=0x7f8334009bd0 tx=0x7f833405c520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:49.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338069320 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 rx=0x7f8334009bd0 tx=0x7f833405c520).stop 2021-03-09T09:29:49.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338069320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:49.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338069320 unknown :-1 s=CLOSED pgs=2908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:49.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:49.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:49.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:49.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025a70 con 0x7f834c0f4690 2021-03-09T09:29:49.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380128e0 con 0x7f8338026c80 2021-03-09T09:29:49.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380123f0 con 0x7f8338062ca0 2021-03-09T09:29:49.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:29:49.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:49.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:29:49.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:49.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:49.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:49.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:49.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:49.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:49.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338018580 con 0x7f8338026c80 2021-03-09T09:29:49.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 rx=0x7f833c05f7d0 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:49.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338026c80 2021-03-09T09:29:49.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.784+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d2d0 con 0x7f8338026c80 2021-03-09T09:29:49.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:49.792+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338026c80 2021-03-09T09:29:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:29:49 smithi146 bash[13136]: debug 2021-03-09T09:29:49.782+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8124 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:29:50.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:50.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2021-03-09T09:29:50.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:50.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:50.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:50.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c039000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:50.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:50.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c039000).stop 2021-03-09T09:29:50.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:50.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T09:29:50.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:50.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:50.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:50.636+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:50.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:50.636+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833801cbc0 0x7f8338063f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:50.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:50.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f834400de80 con 0x7f833801cbc0 2021-03-09T09:29:50.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:50.636+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833801cbc0 0x7f8338063f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:50.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:50.636+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833801cbc0 0x7f8338063f80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f8334004210 tx=0x7f833405c520).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:54.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:29:54.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:54.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 rx=0x7f833c05f7d0 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:54.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 rx=0x7f833c05f7d0 tx=0x7f833c04e870).stop 2021-03-09T09:29:54.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:54.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=CLOSED pgs=2917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:54.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:54.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:54.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:54.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b2c0 con 0x7f834c0f4690 2021-03-09T09:29:54.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e120 con 0x7f8338006a70 2021-03-09T09:29:54.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008120 con 0x7f8338062ca0 2021-03-09T09:29:54.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:29:54.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:54.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:29:54.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:54.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:54.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:54.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:54.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:54.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:54.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026540 con 0x7f8338006a70 2021-03-09T09:29:54.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 rx=0x7f833c040400 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:54.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c069960 con 0x7f8338006a70 2021-03-09T09:29:54.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.788+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8338006a70 2021-03-09T09:29:54.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:54.792+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c022730 con 0x7f8338006a70 2021-03-09T09:29:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:29:54 smithi146 bash[13136]: debug 2021-03-09T09:29:54.786+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8149 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:29:59.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:29:59.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:59.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 rx=0x7f833c040400 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:59.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 rx=0x7f833c040400 tx=0x7f833c04e870).stop 2021-03-09T09:29:59.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:59.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=2926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:59.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:59.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:59.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:29:59.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:59.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:59.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025a70 con 0x7f834c0f4690 2021-03-09T09:29:59.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bda0 con 0x7f8338026c80 2021-03-09T09:29:59.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e160 con 0x7f8338062ca0 2021-03-09T09:29:59.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:59.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:29:59.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:59.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:59.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:59.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:59.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:59.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338018580 con 0x7f8338026c80 2021-03-09T09:29:59.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.787+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 rx=0x7f833c055040 tx=0x7f833c04e870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:59.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c040240 con 0x7f8338026c80 2021-03-09T09:29:59.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0254c0 con 0x7f8338026c80 2021-03-09T09:29:59.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:29:59.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c025b70 con 0x7f8338026c80 2021-03-09T09:30:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:29:59 smithi146 bash[13136]: debug 2021-03-09T09:29:59.786+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8174 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:30:04.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:30:04.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:04.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 rx=0x7f833c055040 tx=0x7f833c04e870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:04.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 rx=0x7f833c055040 tx=0x7f833c04e870).stop 2021-03-09T09:30:04.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833807dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:04.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833807dc80 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:04.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:04.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:04.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:30:04.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:04.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:04.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025a70 con 0x7f834c0f4690 2021-03-09T09:30:04.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065fe0 con 0x7f8338006a70 2021-03-09T09:30:04.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005250 con 0x7f8338062ca0 2021-03-09T09:30:04.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:30:04.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:04.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:04.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:04.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:04.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:04.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:04.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338021c10 con 0x7f8338006a70 2021-03-09T09:30:04.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c024000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:04.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c035820 con 0x7f8338006a70 2021-03-09T09:30:04.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c040ce0 con 0x7f8338006a70 2021-03-09T09:30:04.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:04.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0547c0 con 0x7f8338006a70 2021-03-09T09:30:05.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:30:04 smithi146 bash[13136]: debug 2021-03-09T09:30:04.786+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8200 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:30:09.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:30:09.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:09.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c024000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:09.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c024000).stop 2021-03-09T09:30:09.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:09.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=2943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:09.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:09.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:09.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:09.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800bf00 con 0x7f834c0f4690 2021-03-09T09:30:09.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025a70 con 0x7f8338026c80 2021-03-09T09:30:09.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015770 con 0x7f8338062ca0 2021-03-09T09:30:09.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:30:09.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:09.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:30:09.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:09.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:09.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.795+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:09.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.795+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:09.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.795+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:09.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.795+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:09.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.795+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338018580 con 0x7f8338026c80 2021-03-09T09:30:09.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.795+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806c690 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 rx=0x7f8338021720 tx=0x7f833c024000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:09.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c060260 con 0x7f8338026c80 2021-03-09T09:30:09.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c040ce0 con 0x7f8338026c80 2021-03-09T09:30:09.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:09.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f8338026c80 2021-03-09T09:30:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:30:09 smithi146 bash[13136]: debug 2021-03-09T09:30:09.790+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8223 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:30:14.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:30:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806c690 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 rx=0x7f8338021720 tx=0x7f833c024000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806c690 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 rx=0x7f8338021720 tx=0x7f833c024000).stop 2021-03-09T09:30:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833806c690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833806c690 unknown :-1 s=CLOSED pgs=2952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:30:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:14.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f720 con 0x7f834c0f4690 2021-03-09T09:30:14.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008160 con 0x7f8338006a70 2021-03-09T09:30:14.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025a70 con 0x7f8338062ca0 2021-03-09T09:30:14.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:30:14.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:14.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:14.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:14.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:14.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:14.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:14.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.795+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338069da0 con 0x7f8338006a70 2021-03-09T09:30:14.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801cfa0 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 rx=0x7f83380655a0 tx=0x7f833c024000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:14.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c056210 con 0x7f8338006a70 2021-03-09T09:30:14.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8338006a70 2021-03-09T09:30:14.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:14.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c034200 con 0x7f8338006a70 2021-03-09T09:30:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:30:14 smithi146 bash[13136]: debug 2021-03-09T09:30:14.793+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8249 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:30:19.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833801cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:30:19.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833801cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:19.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801cfa0 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 rx=0x7f83380655a0 tx=0x7f833c024000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:19.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801cfa0 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 rx=0x7f83380655a0 tx=0x7f833c024000).stop 2021-03-09T09:30:19.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833801cfa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:19.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801cfa0 unknown :-1 s=CLOSED pgs=2962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:19.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:19.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:19.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:19.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016a70 con 0x7f834c0f4690 2021-03-09T09:30:19.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800bf00 con 0x7f8338026c80 2021-03-09T09:30:19.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ab0 con 0x7f8338062ca0 2021-03-09T09:30:19.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:30:19.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:19.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:30:19.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:19.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.795+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:19.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.799+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:19.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:19.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.799+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:19.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:19.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.799+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380068e0 con 0x7f8338026c80 2021-03-09T09:30:19.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800ac30 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 rx=0x7f833c070050 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:19.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c02c040 con 0x7f8338026c80 2021-03-09T09:30:19.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c077070 con 0x7f8338026c80 2021-03-09T09:30:19.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:19.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00d280 con 0x7f8338026c80 2021-03-09T09:30:20.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:30:19 smithi146 bash[13136]: debug 2021-03-09T09:30:19.793+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8274 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:30:24.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:30:24.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:24.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800ac30 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 rx=0x7f833c070050 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:24.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800ac30 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 rx=0x7f833c070050 tx=0x7f833c00b040).stop 2021-03-09T09:30:24.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833800ac30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:24.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833800ac30 unknown :-1 s=CLOSED pgs=2969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:24.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:24.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:24.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:30:24.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:30:24.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:24.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:24.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:24.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068c40 con 0x7f834c0f4690 2021-03-09T09:30:24.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014ce0 con 0x7f8338006a70 2021-03-09T09:30:24.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b2c0 con 0x7f8338062ca0 2021-03-09T09:30:24.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:24.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:24.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:24.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:24.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:24.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338018580 con 0x7f8338006a70 2021-03-09T09:30:24.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801cfa0 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 rx=0x7f833c029890 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:24.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0294a0 con 0x7f8338006a70 2021-03-09T09:30:24.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8338006a70 2021-03-09T09:30:24.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:24.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05f200 con 0x7f8338006a70 2021-03-09T09:30:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:30:24 smithi146 bash[13136]: debug 2021-03-09T09:30:24.797+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8300 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:30:29.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.799+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833801cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:30:29.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.799+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833801cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:29.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801cfa0 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 rx=0x7f833c029890 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:29.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801cfa0 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 rx=0x7f833c029890 tx=0x7f833c00b040).stop 2021-03-09T09:30:29.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833801cfa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:29.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801cfa0 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:29.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:29.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:29.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:29.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.799+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:30:29.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.799+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:29.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:29.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800bf00 con 0x7f834c0f4690 2021-03-09T09:30:29.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062970 con 0x7f8338026c80 2021-03-09T09:30:29.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8338062ca0 2021-03-09T09:30:29.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.803+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:30:29.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:29.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.803+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:29.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:29.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.803+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:29.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:29.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.803+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026540 con 0x7f8338026c80 2021-03-09T09:30:29.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802d800 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:29.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.807+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c056210 con 0x7f8338026c80 2021-03-09T09:30:29.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.807+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c069960 con 0x7f8338026c80 2021-03-09T09:30:29.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:29.807+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06a2d0 con 0x7f8338026c80 2021-03-09T09:30:30.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:30:29 smithi146 bash[13136]: debug 2021-03-09T09:30:29.797+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8324 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:30:34.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.799+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833802d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:30:34.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.799+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833802d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:34.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802d800 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:34.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802d800 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c00b040).stop 2021-03-09T09:30:34.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 msgr2=0x7f833802d800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:34.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338026c80 0x7f833802d800 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:34.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:34.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:34.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338027990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:34.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014f20 con 0x7f834c0f4690 2021-03-09T09:30:34.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f720 con 0x7f8338006a70 2021-03-09T09:30:34.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012aa0 con 0x7f8338062ca0 2021-03-09T09:30:34.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.803+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:30:34.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:34.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.803+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:30:34.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.803+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:34.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338027990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:34.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.803+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:34.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:34.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.803+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:34.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:34.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.803+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380068e0 con 0x7f8338006a70 2021-03-09T09:30:34.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338027990 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 rx=0x7f833c025870 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:34.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c069960 con 0x7f8338006a70 2021-03-09T09:30:34.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c040ce0 con 0x7f8338006a70 2021-03-09T09:30:34.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:34.807+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c071240 con 0x7f8338006a70 2021-03-09T09:30:35.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:30:34 smithi146 bash[13136]: debug 2021-03-09T09:30:34.797+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8349 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:30:39.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338027990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:30:39.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338027990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:39.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338027990 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 rx=0x7f833c025870 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:39.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338027990 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 rx=0x7f833c025870 tx=0x7f833c00b040).stop 2021-03-09T09:30:39.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338027990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:39.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338027990 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:39.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:39.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:39.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:39.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021e60 con 0x7f834c0f4690 2021-03-09T09:30:39.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064ed0 con 0x7f833802d7b0 2021-03-09T09:30:39.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f720 con 0x7f8338062ca0 2021-03-09T09:30:39.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:30:39.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:39.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:30:39.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:39.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:39.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.807+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:39.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.807+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:39.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.807+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:39.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.807+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:39.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.807+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f833802d7b0 2021-03-09T09:30:39.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.807+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802db90 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 rx=0x7f833c070050 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:39.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.807+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c069960 con 0x7f833802d7b0 2021-03-09T09:30:39.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.807+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:30:39.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:39.811+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c040970 con 0x7f833802d7b0 2021-03-09T09:30:40.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:30:39 smithi146 bash[13136]: debug 2021-03-09T09:30:39.801+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8374 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:30:44.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.802+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:30:44.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.802+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:44.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.802+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802db90 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 rx=0x7f833c070050 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:44.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.802+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802db90 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 rx=0x7f833c070050 tx=0x7f833c00b040).stop 2021-03-09T09:30:44.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.802+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802db90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:44.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.802+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802db90 unknown :-1 s=CLOSED pgs=3004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:44.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.802+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:44.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.802+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:44.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.802+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:44.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.802+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018170 con 0x7f834c0f4690 2021-03-09T09:30:44.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.802+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800cfa0 con 0x7f8338006a70 2021-03-09T09:30:44.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.802+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012aa0 con 0x7f8338062ca0 2021-03-09T09:30:44.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.806+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:30:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.806+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.806+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:30:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.806+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.806+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.806+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:44.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.806+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:44.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.806+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:44.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.806+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:44.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.806+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026540 con 0x7f8338006a70 2021-03-09T09:30:44.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.806+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833806d150 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 rx=0x7f8338015290 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:44.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344072190 con 0x7f8338006a70 2021-03-09T09:30:44.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072190 con 0x7f8338006a70 2021-03-09T09:30:44.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:44.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400d450 con 0x7f8338006a70 2021-03-09T09:30:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:30:44 smithi146 bash[13136]: debug 2021-03-09T09:30:44.801+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8400 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:30:49.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:30:49.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:49.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833806d150 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 rx=0x7f8338015290 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:49.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833806d150 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 rx=0x7f8338015290 tx=0x7f834404c330).stop 2021-03-09T09:30:49.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833806d150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:49.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833806d150 unknown :-1 s=CLOSED pgs=3012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:49.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:49.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:49.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:30:49.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:49.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:49.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f834c0f4690 2021-03-09T09:30:49.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014f20 con 0x7f833802d7b0 2021-03-09T09:30:49.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012aa0 con 0x7f8338062ca0 2021-03-09T09:30:49.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:30:49.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:49.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.806+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:49.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.810+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:49.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.810+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:49.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.810+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:49.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.810+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:49.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.810+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f833802d7b0 2021-03-09T09:30:49.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.810+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801cfa0 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:49.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344054190 con 0x7f833802d7b0 2021-03-09T09:30:49.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f833802d7b0 2021-03-09T09:30:49.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:49.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834406b610 con 0x7f833802d7b0 2021-03-09T09:30:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:30:49 smithi146 bash[13136]: debug 2021-03-09T09:30:49.805+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8424 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:30:54.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.806+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:30:54.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.806+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:54.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.806+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801cfa0 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:54.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.806+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801cfa0 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f834404c330).stop 2021-03-09T09:30:54.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.806+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801cfa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:54.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.806+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801cfa0 unknown :-1 s=CLOSED pgs=3023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:54.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.806+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:54.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.806+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:54.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.806+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:54.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.806+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380123f0 con 0x7f834c0f4690 2021-03-09T09:30:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.806+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012aa0 con 0x7f8338006a70 2021-03-09T09:30:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.806+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8338062ca0 2021-03-09T09:30:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.806+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:30:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.806+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.806+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:30:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.806+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.810+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.810+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.810+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.810+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.810+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.810+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802d5b0 con 0x7f8338006a70 2021-03-09T09:30:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.810+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023340 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 rx=0x7f833c0707a0 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05c380 con 0x7f8338006a70 2021-03-09T09:30:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f020 con 0x7f8338006a70 2021-03-09T09:30:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:54.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338006a70 2021-03-09T09:30:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:30:54 smithi146 bash[13136]: debug 2021-03-09T09:30:54.805+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8449 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:30:59.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338023340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:30:59.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338023340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:59.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023340 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 rx=0x7f833c0707a0 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:59.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023340 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 rx=0x7f833c0707a0 tx=0x7f833c00b040).stop 2021-03-09T09:30:59.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338023340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:59.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023340 unknown :-1 s=CLOSED pgs=3030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:59.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:59.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833806dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:59.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:59.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011530 con 0x7f834c0f4690 2021-03-09T09:30:59.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007320 con 0x7f833802d7b0 2021-03-09T09:30:59.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8338062ca0 2021-03-09T09:30:59.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:30:59.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:59.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833806dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:59.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:30:59.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:59.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:59.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:59.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:59.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:59.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.810+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068bf0 con 0x7f833802d7b0 2021-03-09T09:30:59.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.814+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833806dc50 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:59.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440171c0 con 0x7f833802d7b0 2021-03-09T09:30:59.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344017390 con 0x7f833802d7b0 2021-03-09T09:30:59.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:30:59.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400d450 con 0x7f833802d7b0 2021-03-09T09:31:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:30:59 smithi146 bash[13136]: debug 2021-03-09T09:30:59.809+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8473 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:31:04.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833806dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:31:04.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833806dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:04.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833806dc50 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:04.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833806dc50 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404c330).stop 2021-03-09T09:31:04.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833806dc50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:04.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833806dc50 unknown :-1 s=CLOSED pgs=3038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:04.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:04.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:04.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:04.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012aa0 con 0x7f834c0f4690 2021-03-09T09:31:04.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072780 con 0x7f8338006a70 2021-03-09T09:31:04.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f720 con 0x7f8338062ca0 2021-03-09T09:31:04.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:31:04.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:04.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:31:04.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:04.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.810+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:04.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.814+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:04.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.814+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:04.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.814+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:04.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.814+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:04.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.814+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806fa10 con 0x7f8338006a70 2021-03-09T09:31:04.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.814+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023340 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:04.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c054ea0 con 0x7f8338006a70 2021-03-09T09:31:04.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8338006a70 2021-03-09T09:31:04.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:04.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c040970 con 0x7f8338006a70 2021-03-09T09:31:05.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:31:04 smithi146 bash[13136]: debug 2021-03-09T09:31:04.809+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8500 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:31:09.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338023340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:31:09.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338023340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:09.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023340 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:09.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023340 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c00b040).stop 2021-03-09T09:31:09.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338023340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:09.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023340 unknown :-1 s=CLOSED pgs=3047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:09.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:09.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:09.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:09.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016910 con 0x7f834c0f4690 2021-03-09T09:31:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b1e0 con 0x7f833802d7b0 2021-03-09T09:31:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012aa0 con 0x7f8338062ca0 2021-03-09T09:31:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:31:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:09.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:31:09.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:09.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:09.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:09.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.814+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:09.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.818+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:09.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:09.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.818+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a4f0 con 0x7f833802d7b0 2021-03-09T09:31:09.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800eca0 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:09.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344079a40 con 0x7f833802d7b0 2021-03-09T09:31:09.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344017390 con 0x7f833802d7b0 2021-03-09T09:31:09.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:09.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405cd70 con 0x7f833802d7b0 2021-03-09T09:31:10.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:31:09 smithi146 bash[13136]: debug 2021-03-09T09:31:09.813+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8523 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:31:14.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.814+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833800eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:31:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.814+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833800eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.814+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800eca0 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.814+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800eca0 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f834404c330).stop 2021-03-09T09:31:14.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833800eca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:14.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.814+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800eca0 unknown :-1 s=CLOSED pgs=3056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:14.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.814+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:14.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.814+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:14.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.814+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012aa0 con 0x7f834c0f4690 2021-03-09T09:31:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b1e0 con 0x7f8338006a70 2021-03-09T09:31:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016910 con 0x7f8338062ca0 2021-03-09T09:31:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.814+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:31:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.814+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:14.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.814+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:31:14.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.814+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:14.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:14.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.818+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:14.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:14.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.818+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:14.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:14.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.818+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024850 con 0x7f8338006a70 2021-03-09T09:31:14.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023a30 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 rx=0x7f834400e010 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:14.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405d1d0 con 0x7f8338006a70 2021-03-09T09:31:14.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338006a70 2021-03-09T09:31:14.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:14.826+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344071130 con 0x7f8338006a70 2021-03-09T09:31:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:31:14 smithi146 bash[13136]: debug 2021-03-09T09:31:14.813+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8550 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:31:19.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338023a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:31:19.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338023a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:19.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023a30 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 rx=0x7f834400e010 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:19.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023a30 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 rx=0x7f834400e010 tx=0x7f834404c330).stop 2021-03-09T09:31:19.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338023a30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:19.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338023a30 unknown :-1 s=CLOSED pgs=3067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:19.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:19.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:19.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:19.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016910 con 0x7f834c0f4690 2021-03-09T09:31:19.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024f00 con 0x7f833802d7b0 2021-03-09T09:31:19.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e530 con 0x7f8338062ca0 2021-03-09T09:31:19.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:31:19.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:19.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:31:19.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:19.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:19.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:19.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:19.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:19.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.818+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:19.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.822+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338018580 con 0x7f833802d7b0 2021-03-09T09:31:19.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.822+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801cfa0 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 rx=0x7f833800e470 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:19.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f833802d7b0 2021-03-09T09:31:19.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:31:19.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:19.826+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d210 con 0x7f833802d7b0 2021-03-09T09:31:20.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:31:19 smithi146 bash[13136]: debug 2021-03-09T09:31:19.817+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8573 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:31:24.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:31:24.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:24.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801cfa0 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 rx=0x7f833800e470 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:24.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801cfa0 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 rx=0x7f833800e470 tx=0x7f833c00b040).stop 2021-03-09T09:31:24.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801cfa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:24.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801cfa0 unknown :-1 s=CLOSED pgs=3073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:24.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:24.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:24.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:24.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b1e0 con 0x7f834c0f4690 2021-03-09T09:31:24.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012aa0 con 0x7f8338006a70 2021-03-09T09:31:24.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016910 con 0x7f8338062ca0 2021-03-09T09:31:24.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:31:24.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:24.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:31:24.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:24.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:24.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.822+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:24.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.822+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:24.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.822+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:24.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.822+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:24.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.822+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024850 con 0x7f8338006a70 2021-03-09T09:31:24.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.822+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801d490 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 rx=0x7f8344028ee0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:24.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344032930 con 0x7f8338006a70 2021-03-09T09:31:24.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405d1d0 con 0x7f8338006a70 2021-03-09T09:31:24.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:24.826+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338006a70 2021-03-09T09:31:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:31:24 smithi146 bash[13136]: debug 2021-03-09T09:31:24.816+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8600 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:31:29.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833801d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:31:29.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833801d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:29.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801d490 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 rx=0x7f8344028ee0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:29.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801d490 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 rx=0x7f8344028ee0 tx=0x7f834404c330).stop 2021-03-09T09:31:29.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f833801d490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:29.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f833801d490 unknown :-1 s=CLOSED pgs=3082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:29.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:29.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:29.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:29.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016910 con 0x7f834c0f4690 2021-03-09T09:31:29.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018000 con 0x7f833802d7b0 2021-03-09T09:31:29.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ce40 con 0x7f8338062ca0 2021-03-09T09:31:29.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:31:29.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:29.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:31:29.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:29.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:29.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:29.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:29.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:29.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:29.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.821+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f833802d7b0 2021-03-09T09:31:29.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.825+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801d980 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 rx=0x7f833c041000 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:29.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.825+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f833802d7b0 2021-03-09T09:31:29.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.825+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d1c0 con 0x7f833802d7b0 2021-03-09T09:31:29.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:29.829+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c040970 con 0x7f833802d7b0 2021-03-09T09:31:30.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:31:29 smithi146 bash[13136]: debug 2021-03-09T09:31:29.820+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8623 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:31:34.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.821+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:31:34.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.821+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:34.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.821+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801d980 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 rx=0x7f833c041000 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:34.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.821+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801d980 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 rx=0x7f833c041000 tx=0x7f833c00b040).stop 2021-03-09T09:31:34.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.821+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801d980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:34.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.821+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801d980 unknown :-1 s=CLOSED pgs=3090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:34.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.821+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:34.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.821+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:34.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.821+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:34.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.821+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b940 con 0x7f834c0f4690 2021-03-09T09:31:34.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.821+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015f80 con 0x7f8338006a70 2021-03-09T09:31:34.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.821+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016910 con 0x7f8338062ca0 2021-03-09T09:31:34.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.821+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:31:34.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.821+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:34.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.821+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:31:34.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.821+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:34.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.825+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:34.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.825+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:34.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.825+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:34.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.825+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:34.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.825+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:34.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.825+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024850 con 0x7f8338006a70 2021-03-09T09:31:34.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.825+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338025e80 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 rx=0x7f833c05fe20 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:34.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.829+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c025d70 con 0x7f8338006a70 2021-03-09T09:31:34.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.829+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8338006a70 2021-03-09T09:31:34.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:34.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0701d0 con 0x7f8338006a70 2021-03-09T09:31:35.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:31:34 smithi146 bash[13136]: debug 2021-03-09T09:31:34.820+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8650 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:31:39.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:31:39.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:39.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338025e80 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 rx=0x7f833c05fe20 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:39.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338025e80 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 rx=0x7f833c05fe20 tx=0x7f833c00b040).stop 2021-03-09T09:31:39.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:39.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006a70 0x7f8338025e80 unknown :-1 s=CLOSED pgs=3102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:39.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:39.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:39.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:39.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800eb30 con 0x7f834c0f4690 2021-03-09T09:31:39.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b800 con 0x7f833802d7b0 2021-03-09T09:31:39.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016910 con 0x7f8338062ca0 2021-03-09T09:31:39.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:31:39.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:39.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:31:39.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:39.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.825+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:39.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:39.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:39.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:39.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:39.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380241b0 con 0x7f833802d7b0 2021-03-09T09:31:39.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801d9c0 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 rx=0x7f833c060a70 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:39.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c058ee0 con 0x7f833802d7b0 2021-03-09T09:31:39.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c054ee0 con 0x7f833802d7b0 2021-03-09T09:31:39.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:39.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c035c20 con 0x7f833802d7b0 2021-03-09T09:31:40.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:31:39 smithi146 bash[13136]: debug 2021-03-09T09:31:39.824+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8673 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:31:44.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.825+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:31:44.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.825+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:44.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.825+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801d9c0 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 rx=0x7f833c060a70 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:44.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.825+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801d9c0 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 rx=0x7f833c060a70 tx=0x7f833c00b040).stop 2021-03-09T09:31:44.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.825+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801d9c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:44.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.825+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801d9c0 unknown :-1 s=CLOSED pgs=3109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:44.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.825+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:44.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.825+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:44.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.825+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:44.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.825+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063ba0 con 0x7f834c0f4690 2021-03-09T09:31:44.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.825+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806ca70 con 0x7f833800eed0 2021-03-09T09:31:44.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.825+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015a10 con 0x7f8338062ca0 2021-03-09T09:31:44.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.825+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:31:44.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.825+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:44.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.825+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:31:44.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.825+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:44.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.829+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:44.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.829+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:44.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.829+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:44.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.829+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:44.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.829+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:44.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.829+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802bb60 con 0x7f833800eed0 2021-03-09T09:31:44.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.829+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801deb0 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:44.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.829+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344054c60 con 0x7f833800eed0 2021-03-09T09:31:44.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.829+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344054dc0 con 0x7f833800eed0 2021-03-09T09:31:44.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:44.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344054c60 con 0x7f833800eed0 2021-03-09T09:31:45.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:31:44 smithi146 bash[13136]: debug 2021-03-09T09:31:44.824+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8700 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:31:49.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.825+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833801deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:31:49.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.825+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833801deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:49.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.825+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801deb0 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:49.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.825+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801deb0 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f834404c330).stop 2021-03-09T09:31:49.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.825+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833801deb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:49.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.825+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801deb0 unknown :-1 s=CLOSED pgs=3117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:49.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.825+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:49.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.825+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:49.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.825+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:49.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.825+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b940 con 0x7f834c0f4690 2021-03-09T09:31:49.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.825+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016910 con 0x7f833802d7b0 2021-03-09T09:31:49.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.825+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ee0 con 0x7f8338062ca0 2021-03-09T09:31:49.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:31:49.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:49.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:31:49.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:49.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.829+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.829+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.829+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.829+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.829+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.829+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f833802d7b0 2021-03-09T09:31:49.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.829+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:49.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834407ad10 con 0x7f833802d7b0 2021-03-09T09:31:49.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033040 con 0x7f833802d7b0 2021-03-09T09:31:49.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:49.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344017a20 con 0x7f833802d7b0 2021-03-09T09:31:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:31:49 smithi146 bash[13136]: debug 2021-03-09T09:31:49.828+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8723 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:31:54.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:31:54.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:54.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:54.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404c330).stop 2021-03-09T09:31:54.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:54.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 unknown :-1 s=CLOSED pgs=3127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d380 con 0x7f834c0f4690 2021-03-09T09:31:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066190 con 0x7f833800eed0 2021-03-09T09:31:54.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063ba0 con 0x7f8338062ca0 2021-03-09T09:31:54.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:31:54.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:54.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:31:54.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:54.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:54.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338023dd0 con 0x7f833800eed0 2021-03-09T09:31:54.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.833+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801e1f0 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 rx=0x7f833c034c20 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:54.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c025ed0 con 0x7f833800eed0 2021-03-09T09:31:54.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c040eb0 con 0x7f833800eed0 2021-03-09T09:31:54.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:54.837+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f833800eed0 2021-03-09T09:31:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:31:54 smithi146 bash[13136]: debug 2021-03-09T09:31:54.828+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8750 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:31:59.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833801e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:31:59.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833801e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:59.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801e1f0 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 rx=0x7f833c034c20 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:59.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801e1f0 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 rx=0x7f833c034c20 tx=0x7f833c00b040).stop 2021-03-09T09:31:59.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.829+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833801e1f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:59.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.829+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801e1f0 unknown :-1 s=CLOSED pgs=3134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:59.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.829+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380656c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:59.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.833+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:59.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.833+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:59.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015a10 con 0x7f834c0f4690 2021-03-09T09:31:59.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066190 con 0x7f833802d7b0 2021-03-09T09:31:59.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8338062ca0 2021-03-09T09:31:59.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.833+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:31:59.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.833+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:31:59.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.833+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:59.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.833+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:59.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.833+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380656c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:59.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.837+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:59.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.837+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:59.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.837+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:59.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.837+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:59.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.837+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338069c00 con 0x7f833802d7b0 2021-03-09T09:31:59.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.837+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380656c0 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 rx=0x7f833c056c50 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:59.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.837+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c078c40 con 0x7f833802d7b0 2021-03-09T09:31:59.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.837+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:31:59.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:31:59.841+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c060cd0 con 0x7f833802d7b0 2021-03-09T09:32:00.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:31:59 smithi146 bash[13136]: debug 2021-03-09T09:31:59.828+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8773 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:32:04.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380656c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:32:04.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380656c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:04.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380656c0 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 rx=0x7f833c056c50 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:04.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380656c0 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 rx=0x7f833c056c50 tx=0x7f833c00b040).stop 2021-03-09T09:32:04.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380656c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:04.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380656c0 unknown :-1 s=CLOSED pgs=3143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:04.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:04.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806fb30 con 0x7f834c0f4690 2021-03-09T09:32:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010ab0 con 0x7f833800eed0 2021-03-09T09:32:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015a10 con 0x7f8338062ca0 2021-03-09T09:32:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:32:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:32:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.833+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:04.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.837+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:04.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.837+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:04.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.837+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:04.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.837+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:04.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.837+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380241b0 con 0x7f833800eed0 2021-03-09T09:32:04.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.837+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801e9f0 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 rx=0x7f833c0128d0 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:04.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.837+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01ae60 con 0x7f833800eed0 2021-03-09T09:32:04.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.837+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f030 con 0x7f833800eed0 2021-03-09T09:32:04.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:04.841+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055310 con 0x7f833800eed0 2021-03-09T09:32:05.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:32:04 smithi146 bash[13136]: debug 2021-03-09T09:32:04.832+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8800 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:32:09.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.833+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833801e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:32:09.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.833+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833801e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:09.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.833+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801e9f0 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 rx=0x7f833c0128d0 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:09.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.833+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801e9f0 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 rx=0x7f833c0128d0 tx=0x7f833c00b040).stop 2021-03-09T09:32:09.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833801e9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:09.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.833+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801e9f0 unknown :-1 s=CLOSED pgs=3151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:09.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.833+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:09.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.833+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:09.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.833+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:09.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005960 con 0x7f834c0f4690 2021-03-09T09:32:09.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800bed0 con 0x7f833802d7b0 2021-03-09T09:32:09.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bc00 con 0x7f8338062ca0 2021-03-09T09:32:09.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.833+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:32:09.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.833+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:09.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.837+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:32:09.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.837+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:09.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.837+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:09.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.837+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:09.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.837+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:09.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.837+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:09.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.837+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:09.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.837+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338011530 con 0x7f833802d7b0 2021-03-09T09:32:09.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.841+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 rx=0x7f833c077580 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:09.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.841+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0231e0 con 0x7f833802d7b0 2021-03-09T09:32:09.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.841+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:32:09.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:09.845+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01a270 con 0x7f833802d7b0 2021-03-09T09:32:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:32:09 smithi146 bash[13136]: debug 2021-03-09T09:32:09.836+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8822 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:32:14.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:32:14.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:14.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 rx=0x7f833c077580 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:14.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 rx=0x7f833c077580 tx=0x7f833c00b040).stop 2021-03-09T09:32:14.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:14.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 unknown :-1 s=CLOSED pgs=3164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:14.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:14.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338010c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:14.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:14.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022010 con 0x7f834c0f4690 2021-03-09T09:32:14.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015a10 con 0x7f833800eed0 2021-03-09T09:32:14.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ce40 con 0x7f8338062ca0 2021-03-09T09:32:14.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:32:14.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338010c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:14.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:14.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:32:14.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:14.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:14.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:14.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:14.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:14.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.836+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802c9e0 con 0x7f833800eed0 2021-03-09T09:32:14.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.840+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338010c50 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:14.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404a8d0 con 0x7f833800eed0 2021-03-09T09:32:14.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033040 con 0x7f833800eed0 2021-03-09T09:32:14.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:14.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f833800eed0 2021-03-09T09:32:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:32:14 smithi146 bash[13136]: debug 2021-03-09T09:32:14.840+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8849 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:32:19.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338010c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:32:19.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338010c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:19.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338010c50 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:19.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338010c50 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404c330).stop 2021-03-09T09:32:19.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338010c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:19.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338010c50 unknown :-1 s=CLOSED pgs=3168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:19.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:19.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:19.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:19.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f834c0f4690 2021-03-09T09:32:19.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b970 con 0x7f833802d7b0 2021-03-09T09:32:19.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806ca70 con 0x7f8338062ca0 2021-03-09T09:32:19.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:32:19.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:32:19.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:19.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:19.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:19.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:19.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:19.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:19.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:19.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338023dd0 con 0x7f833802d7b0 2021-03-09T09:32:19.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801be90 secure :-1 s=READY pgs=3177 cs=0 l=1 rev1=1 rx=0x7f833c0777b0 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:19.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0771f0 con 0x7f833802d7b0 2021-03-09T09:32:19.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:32:19.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:19.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055310 con 0x7f833802d7b0 2021-03-09T09:32:20.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:32:19 smithi146 bash[13136]: debug 2021-03-09T09:32:19.840+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8873 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:32:24.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:32:24.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:24.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801be90 secure :-1 s=READY pgs=3177 cs=0 l=1 rev1=1 rx=0x7f833c0777b0 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:24.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801be90 secure :-1 s=READY pgs=3177 cs=0 l=1 rev1=1 rx=0x7f833c0777b0 tx=0x7f833c00b040).stop 2021-03-09T09:32:24.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801be90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:24.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801be90 unknown :-1 s=CLOSED pgs=3177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:24.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:24.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:24.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:24.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cbe0 con 0x7f834c0f4690 2021-03-09T09:32:24.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010ab0 con 0x7f833800eed0 2021-03-09T09:32:24.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022010 con 0x7f8338062ca0 2021-03-09T09:32:24.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:32:24.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:32:24.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:24.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:24.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:24.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:24.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:24.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:24.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:24.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005940 con 0x7f833800eed0 2021-03-09T09:32:24.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 secure :-1 s=READY pgs=3185 cs=0 l=1 rev1=1 rx=0x7f833c069040 tx=0x7f833c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:24.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c069960 con 0x7f833800eed0 2021-03-09T09:32:24.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c069960 con 0x7f833800eed0 2021-03-09T09:32:24.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:24.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05fbd0 con 0x7f833800eed0 2021-03-09T09:32:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:32:24 smithi146 bash[13136]: debug 2021-03-09T09:32:24.844+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8899 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:32:29.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.840+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:32:29.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.840+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:29.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.840+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 secure :-1 s=READY pgs=3185 cs=0 l=1 rev1=1 rx=0x7f833c069040 tx=0x7f833c00b040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:29.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.840+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 secure :-1 s=READY pgs=3185 cs=0 l=1 rev1=1 rx=0x7f833c069040 tx=0x7f833c00b040).stop 2021-03-09T09:32:29.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:29.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.840+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 unknown :-1 s=CLOSED pgs=3185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:29.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.840+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:29.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.840+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:29.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.840+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:29.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f834c0f4690 2021-03-09T09:32:29.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015a10 con 0x7f833802d7b0 2021-03-09T09:32:29.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022010 con 0x7f8338062ca0 2021-03-09T09:32:29.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.840+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:32:29.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.840+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:29.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.840+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:32:29.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.840+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:29.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:29.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:29.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:29.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:29.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:29.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800cdf0 con 0x7f833802d7b0 2021-03-09T09:32:29.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802c100 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 rx=0x7f833c040c80 tx=0x7f833c0533a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:29.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00f300 con 0x7f833802d7b0 2021-03-09T09:32:29.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c069960 con 0x7f833802d7b0 2021-03-09T09:32:29.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:29.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f833802d7b0 2021-03-09T09:32:30.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:32:29 smithi146 bash[13136]: debug 2021-03-09T09:32:29.843+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8923 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:32:34.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:32:34.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:34.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802c100 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 rx=0x7f833c040c80 tx=0x7f833c0533a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:34.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802c100 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 rx=0x7f833c040c80 tx=0x7f833c0533a0).stop 2021-03-09T09:32:34.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802c100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:34.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802c100 unknown :-1 s=CLOSED pgs=3194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:34.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:34.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:34.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:34.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cc70 con 0x7f834c0f4690 2021-03-09T09:32:34.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008ae0 con 0x7f833800eed0 2021-03-09T09:32:34.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f8338062ca0 2021-03-09T09:32:34.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:32:34.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:34.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:32:34.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:34.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.844+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:34.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.848+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:34.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.848+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:34.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.848+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:34.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.848+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:34.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.848+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f833800eed0 2021-03-09T09:32:34.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.848+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 rx=0x7f834405df90 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:34.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405dd90 con 0x7f833800eed0 2021-03-09T09:32:34.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833800eed0 2021-03-09T09:32:34.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:34.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404c8f0 con 0x7f833800eed0 2021-03-09T09:32:35.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:32:34 smithi146 bash[13136]: debug 2021-03-09T09:32:34.847+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8949 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:32:39.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.844+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:32:39.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.844+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:39.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.844+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 rx=0x7f834405df90 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:39.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.844+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 rx=0x7f834405df90 tx=0x7f834404c330).stop 2021-03-09T09:32:39.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:39.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.844+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 unknown :-1 s=CLOSED pgs=3204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:39.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.844+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:39.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.844+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338069d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:39.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.844+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:39.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f834c0f4690 2021-03-09T09:32:39.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022010 con 0x7f833802d7b0 2021-03-09T09:32:39.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b1e0 con 0x7f8338062ca0 2021-03-09T09:32:39.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:32:39.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:39.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:32:39.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:39.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.848+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338069d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:39.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.848+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:39.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.848+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:39.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.848+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:39.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.848+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:39.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.848+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:32:39.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.848+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338069d90 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:39.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344037630 con 0x7f833802d7b0 2021-03-09T09:32:39.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833802d7b0 2021-03-09T09:32:39.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:39.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834401f470 con 0x7f833802d7b0 2021-03-09T09:32:40.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:32:39 smithi146 bash[13136]: debug 2021-03-09T09:32:39.847+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8972 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:32:44.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338069d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:32:44.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338069d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:44.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338069d90 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:44.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338069d90 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404c330).stop 2021-03-09T09:32:44.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338069d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:44.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338069d90 unknown :-1 s=CLOSED pgs=3213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:44.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:44.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338067ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:44.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:44.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cc70 con 0x7f834c0f4690 2021-03-09T09:32:44.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f833800eed0 2021-03-09T09:32:44.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d660 con 0x7f8338062ca0 2021-03-09T09:32:44.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:32:44.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:44.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:32:44.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:44.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.848+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338067ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:44.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.852+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:44.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.852+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:44.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.852+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:44.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.852+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:44.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.852+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800cdf0 con 0x7f833800eed0 2021-03-09T09:32:44.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.852+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338067ee0 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 rx=0x7f834401fff0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:44.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400fb80 con 0x7f833800eed0 2021-03-09T09:32:44.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400f9a0 con 0x7f833800eed0 2021-03-09T09:32:44.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:44.856+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404b890 con 0x7f833800eed0 2021-03-09T09:32:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:32:44 smithi146 bash[13136]: debug 2021-03-09T09:32:44.851+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 8998 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:32:49.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:32:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338067ee0 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 rx=0x7f834401fff0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338067ee0 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 rx=0x7f834401fff0 tx=0x7f834404c330).stop 2021-03-09T09:32:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338067ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338067ee0 unknown :-1 s=CLOSED pgs=3222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:49.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:49.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:49.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:49.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f834c0f4690 2021-03-09T09:32:49.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d660 con 0x7f833802d7b0 2021-03-09T09:32:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800cc60 con 0x7f8338062ca0 2021-03-09T09:32:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:32:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:32:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005940 con 0x7f833802d7b0 2021-03-09T09:32:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 secure :-1 s=READY pgs=3229 cs=0 l=1 rev1=1 rx=0x7f833c018850 tx=0x7f833c0533a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f833802d7b0 2021-03-09T09:32:49.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f833802d7b0 2021-03-09T09:32:49.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:49.856+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833802d7b0 2021-03-09T09:32:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:32:49 smithi146 bash[13136]: debug 2021-03-09T09:32:49.851+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9023 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:32:54.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380134e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:32:54.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380134e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:54.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 secure :-1 s=READY pgs=3229 cs=0 l=1 rev1=1 rx=0x7f833c018850 tx=0x7f833c0533a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:54.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 secure :-1 s=READY pgs=3229 cs=0 l=1 rev1=1 rx=0x7f833c018850 tx=0x7f833c0533a0).stop 2021-03-09T09:32:54.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380134e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:54.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 unknown :-1 s=CLOSED pgs=3229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:54.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338067ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:54.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:54.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:54.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f834c0f4690 2021-03-09T09:32:54.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800cc60 con 0x7f833800eed0 2021-03-09T09:32:54.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ee0 con 0x7f8338062ca0 2021-03-09T09:32:54.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:32:54.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:54.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:32:54.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:54.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.852+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338067ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:54.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.856+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:54.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.856+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:54.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.856+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:54.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.856+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:54.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.856+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833800eed0 2021-03-09T09:32:54.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.856+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338067ee0 secure :-1 s=READY pgs=3243 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c0533a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:54.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.856+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c060260 con 0x7f833800eed0 2021-03-09T09:32:54.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.856+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f833800eed0 2021-03-09T09:32:54.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:54.860+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0792c0 con 0x7f833800eed0 2021-03-09T09:32:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:32:54 smithi146 bash[13136]: debug 2021-03-09T09:32:54.855+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9045 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:32:59.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:32:59.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:59.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338067ee0 secure :-1 s=READY pgs=3243 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c0533a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:59.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338067ee0 secure :-1 s=READY pgs=3243 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c0533a0).stop 2021-03-09T09:32:59.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338067ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:59.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338067ee0 unknown :-1 s=CLOSED pgs=3243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:59.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:59.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:59.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:59.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f834c0f4690 2021-03-09T09:32:59.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b2d0 con 0x7f833802d7b0 2021-03-09T09:32:59.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d380 con 0x7f8338062ca0 2021-03-09T09:32:59.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:32:59.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:59.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:32:59.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:59.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:59.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:59.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:59.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:59.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:59.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801c7d0 con 0x7f833802d7b0 2021-03-09T09:32:59.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.855+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 rx=0x7f833c0185e0 tx=0x7f833c065040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:59.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018650 con 0x7f833802d7b0 2021-03-09T09:32:59.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c069960 con 0x7f833802d7b0 2021-03-09T09:32:59.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:32:59.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05f2e0 con 0x7f833802d7b0 2021-03-09T09:33:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:32:59 smithi146 bash[13136]: debug 2021-03-09T09:32:59.855+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9073 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:33:04.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380134e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:33:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380134e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 rx=0x7f833c0185e0 tx=0x7f833c065040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 rx=0x7f833c0185e0 tx=0x7f833c065040).stop 2021-03-09T09:33:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380134e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 unknown :-1 s=CLOSED pgs=3250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:04.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:04.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:04.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:04.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012b10 con 0x7f834c0f4690 2021-03-09T09:33:04.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f8d0 con 0x7f833800eed0 2021-03-09T09:33:04.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011730 con 0x7f8338062ca0 2021-03-09T09:33:04.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:33:04.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:04.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:33:04.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:04.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:04.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:04.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:04.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:04.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:04.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.859+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e530 con 0x7f833800eed0 2021-03-09T09:33:04.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.863+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 rx=0x7f833c0619e0 tx=0x7f833c065040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:04.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c060260 con 0x7f833800eed0 2021-03-09T09:33:04.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f030 con 0x7f833800eed0 2021-03-09T09:33:04.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:04.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c029b40 con 0x7f833800eed0 2021-03-09T09:33:05.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:33:04 smithi146 bash[13136]: debug 2021-03-09T09:33:04.859+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9098 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:33:09.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.859+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:33:10.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.859+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:10.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.859+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 rx=0x7f833c0619e0 tx=0x7f833c065040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:10.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.859+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 rx=0x7f833c0619e0 tx=0x7f833c065040).stop 2021-03-09T09:33:10.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:10.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.859+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 unknown :-1 s=CLOSED pgs=3255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:10.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.859+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:10.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.859+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:10.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.859+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:10.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d380 con 0x7f834c0f4690 2021-03-09T09:33:10.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806b180 con 0x7f833802d7b0 2021-03-09T09:33:10.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012b10 con 0x7f8338062ca0 2021-03-09T09:33:10.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.859+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:33:10.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.859+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:10.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.859+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:33:10.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.859+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:10.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.863+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:10.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.863+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:10.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.863+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:10.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.863+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:10.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.863+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:10.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.863+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014750 con 0x7f833802d7b0 2021-03-09T09:33:10.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.863+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:10.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f833802d7b0 2021-03-09T09:33:10.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833802d7b0 2021-03-09T09:33:10.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:09.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834406e120 con 0x7f833802d7b0 2021-03-09T09:33:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:33:09 smithi146 bash[13136]: debug 2021-03-09T09:33:09.859+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9123 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:33:14.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380134e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:33:14.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380134e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404c330).stop 2021-03-09T09:33:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380134e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380134e0 unknown :-1 s=CLOSED pgs=3264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802b700 con 0x7f834c0f4690 2021-03-09T09:33:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380289a0 con 0x7f833800eed0 2021-03-09T09:33:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016180 con 0x7f8338062ca0 2021-03-09T09:33:14.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:33:14.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:14.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:33:14.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:14.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.863+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:14.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:14.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:14.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:14.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:14.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833800eed0 2021-03-09T09:33:14.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 rx=0x7f833c01afc0 tx=0x7f833c065040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c061d80 con 0x7f833800eed0 2021-03-09T09:33:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833800eed0 2021-03-09T09:33:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:14.875+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c040970 con 0x7f833800eed0 2021-03-09T09:33:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:33:14 smithi146 bash[13136]: debug 2021-03-09T09:33:14.863+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9148 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:33:19.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.863+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:33:19.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.863+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:19.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.863+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 rx=0x7f833c01afc0 tx=0x7f833c065040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:19.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.863+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 rx=0x7f833c01afc0 tx=0x7f833c065040).stop 2021-03-09T09:33:19.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:19.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.863+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338025e80 unknown :-1 s=CLOSED pgs=3273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:19.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.863+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:19.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.863+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:19.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.863+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:19.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012b10 con 0x7f834c0f4690 2021-03-09T09:33:19.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c9e0 con 0x7f833802d7b0 2021-03-09T09:33:19.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8338062ca0 2021-03-09T09:33:19.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.863+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:33:19.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.863+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:19.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.863+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:33:19.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.863+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:19.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:19.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:19.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:19.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:19.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:19.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801c7d0 con 0x7f833802d7b0 2021-03-09T09:33:19.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019650 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c065040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:19.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f833802d7b0 2021-03-09T09:33:19.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f030 con 0x7f833802d7b0 2021-03-09T09:33:19.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:19.871+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c062ae0 con 0x7f833802d7b0 2021-03-09T09:33:20.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:33:19 smithi146 bash[13136]: debug 2021-03-09T09:33:19.863+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9173 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:33:24.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:33:24.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:24.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019650 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c065040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:24.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019650 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c065040).stop 2021-03-09T09:33:24.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338019650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:24.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019650 unknown :-1 s=CLOSED pgs=3281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:24.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833800cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:24.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:24.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:24.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027780 con 0x7f834c0f4690 2021-03-09T09:33:24.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380287f0 con 0x7f833800eed0 2021-03-09T09:33:24.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d580 con 0x7f8338062ca0 2021-03-09T09:33:24.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:33:24.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:33:24.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:24.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:24.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833800cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:24.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:24.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:24.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:24.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:24.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068770 con 0x7f833800eed0 2021-03-09T09:33:24.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833800cc70 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 rx=0x7f833c061ff0 tx=0x7f833c065040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:24.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.871+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c023410 con 0x7f833800eed0 2021-03-09T09:33:24.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.871+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f030 con 0x7f833800eed0 2021-03-09T09:33:24.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:24.875+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055310 con 0x7f833800eed0 2021-03-09T09:33:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:33:24 smithi146 bash[13136]: debug 2021-03-09T09:33:24.867+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9198 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:33:29.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833800cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:33:29.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833800cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:29.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833800cc70 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 rx=0x7f833c061ff0 tx=0x7f833c065040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:29.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833800cc70 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 rx=0x7f833c061ff0 tx=0x7f833c065040).stop 2021-03-09T09:33:29.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833800cc70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:29.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.867+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833800cc70 unknown :-1 s=CLOSED pgs=3290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:29.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.867+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:29.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.867+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:29.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.867+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:29.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380103c0 con 0x7f834c0f4690 2021-03-09T09:33:29.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d580 con 0x7f833802d7b0 2021-03-09T09:33:29.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027780 con 0x7f8338062ca0 2021-03-09T09:33:29.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:33:29.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:29.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:33:29.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:29.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.871+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:29.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.871+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:29.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.871+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:29.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.871+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:29.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.871+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:29.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.871+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e4b0 con 0x7f833802d7b0 2021-03-09T09:33:29.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.871+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019650 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:29.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.875+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f833802d7b0 2021-03-09T09:33:29.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.875+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072bb0 con 0x7f833802d7b0 2021-03-09T09:33:29.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:29.875+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834406e190 con 0x7f833802d7b0 2021-03-09T09:33:30.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:33:29 smithi146 bash[13136]: debug 2021-03-09T09:33:29.867+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9223 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:33:34.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:33:34.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:34.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019650 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:34.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019650 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404c330).stop 2021-03-09T09:33:34.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338019650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:34.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019650 unknown :-1 s=CLOSED pgs=3299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:34.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:34.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:34.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:34.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014070 con 0x7f834c0f4690 2021-03-09T09:33:34.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380103c0 con 0x7f833800eed0 2021-03-09T09:33:34.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380111d0 con 0x7f8338062ca0 2021-03-09T09:33:34.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:33:34.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:33:34.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:34.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:34.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:34.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:34.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:34.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:34.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:34.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833805eb00 con 0x7f833800eed0 2021-03-09T09:33:34.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.871+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801a3e0 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c065040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:34.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.875+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0546d0 con 0x7f833800eed0 2021-03-09T09:33:34.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.875+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0544f0 con 0x7f833800eed0 2021-03-09T09:33:34.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:34.879+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f833800eed0 2021-03-09T09:33:35.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:33:34 smithi146 bash[13136]: debug 2021-03-09T09:33:34.871+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9248 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:33:39.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.871+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833801a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:33:39.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.871+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833801a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:39.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.871+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801a3e0 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c065040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:39.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.871+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801a3e0 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c065040).stop 2021-03-09T09:33:39.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.871+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833801a3e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:39.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.871+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801a3e0 unknown :-1 s=CLOSED pgs=3307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:39.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.871+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:39.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.871+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:39.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.871+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:39.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.871+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027780 con 0x7f834c0f4690 2021-03-09T09:33:39.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.871+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ed80 con 0x7f833802d7b0 2021-03-09T09:33:39.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.871+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cc70 con 0x7f8338062ca0 2021-03-09T09:33:39.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.871+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:33:39.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.871+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:39.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.871+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:33:39.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.871+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:39.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.875+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:39.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.879+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:39.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.879+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:39.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.879+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:39.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.879+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:39.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.879+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f833802d7b0 2021-03-09T09:33:39.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.883+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 rx=0x7f8344055040 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:39.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.883+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029050 con 0x7f833802d7b0 2021-03-09T09:33:39.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.883+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344032990 con 0x7f833802d7b0 2021-03-09T09:33:39.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:39.883+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344072cc0 con 0x7f833802d7b0 2021-03-09T09:33:40.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:33:39 smithi146 bash[13136]: debug 2021-03-09T09:33:39.870+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9274 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:33:44.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:33:44.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:44.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 rx=0x7f8344055040 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:44.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 rx=0x7f8344055040 tx=0x7f834404c330).stop 2021-03-09T09:33:44.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:44.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338025e80 unknown :-1 s=CLOSED pgs=3318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:44.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:44.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:44.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:44.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380687c0 con 0x7f834c0f4690 2021-03-09T09:33:44.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a910 con 0x7f833800eed0 2021-03-09T09:33:44.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e360 con 0x7f8338062ca0 2021-03-09T09:33:44.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:33:44.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:44.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:44.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:33:44.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:44.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:44.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:44.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:44.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:44.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.874+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833800eed0 2021-03-09T09:33:44.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.878+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801fba0 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 rx=0x7f833c019c50 tx=0x7f833c065040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:44.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c019860 con 0x7f833800eed0 2021-03-09T09:33:44.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833800eed0 2021-03-09T09:33:44.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:44.882+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c019cc0 con 0x7f833800eed0 2021-03-09T09:33:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:33:44 smithi146 bash[13136]: debug 2021-03-09T09:33:44.874+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9299 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:33:49.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833801fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:33:49.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833801fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:49.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801fba0 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 rx=0x7f833c019c50 tx=0x7f833c065040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:49.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801fba0 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 rx=0x7f833c019c50 tx=0x7f833c065040).stop 2021-03-09T09:33:49.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833801fba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:49.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833801fba0 unknown :-1 s=CLOSED pgs=3324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:49.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338065000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024840 con 0x7f834c0f4690 2021-03-09T09:33:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e360 con 0x7f833802d7b0 2021-03-09T09:33:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380082c0 con 0x7f8338062ca0 2021-03-09T09:33:49.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:33:49.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:49.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:33:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.874+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338065000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.878+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.878+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.878+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.878+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.878+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:33:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.878+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338065000 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c065040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:49.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.882+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c058ec0 con 0x7f833802d7b0 2021-03-09T09:33:49.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.882+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:33:49.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:49.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c019820 con 0x7f833802d7b0 2021-03-09T09:33:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:33:49 smithi146 bash[13136]: debug 2021-03-09T09:33:49.874+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9325 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:33:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338065000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:33:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338065000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338065000 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c065040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338065000 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c065040).stop 2021-03-09T09:33:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338065000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338065000 unknown :-1 s=CLOSED pgs=3334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338004780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dae0 con 0x7f834c0f4690 2021-03-09T09:33:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022e70 con 0x7f833800eed0 2021-03-09T09:33:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016cd0 con 0x7f8338062ca0 2021-03-09T09:33:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:33:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:33:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:54.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.878+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338009a10 con 0x7f833800eed0 2021-03-09T09:33:54.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.882+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338004780 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 rx=0x7f833c0095f0 tx=0x7f833c065040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:54.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.882+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f833800eed0 2021-03-09T09:33:54.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.882+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f833800eed0 2021-03-09T09:33:54.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:54.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0204c0 con 0x7f833800eed0 2021-03-09T09:33:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:33:54 smithi146 bash[13136]: debug 2021-03-09T09:33:54.878+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9350 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:33:59.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:33:59.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:59.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338004780 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 rx=0x7f833c0095f0 tx=0x7f833c065040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:59.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338004780 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 rx=0x7f833c0095f0 tx=0x7f833c065040).stop 2021-03-09T09:33:59.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338004780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:59.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338004780 unknown :-1 s=CLOSED pgs=3342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:59.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:59.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338007c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:59.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:59.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dae0 con 0x7f834c0f4690 2021-03-09T09:33:59.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c9e0 con 0x7f833802d7b0 2021-03-09T09:33:59.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027780 con 0x7f8338062ca0 2021-03-09T09:33:59.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:33:59.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:59.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:33:59.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:59.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.878+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338007c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:59.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.882+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:59.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.882+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:59.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.882+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:59.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.882+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:59.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.882+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338063df0 con 0x7f833802d7b0 2021-03-09T09:33:59.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.882+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338007c40 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:59.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f833802d7b0 2021-03-09T09:33:59.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033030 con 0x7f833802d7b0 2021-03-09T09:33:59.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:33:59.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834406b3f0 con 0x7f833802d7b0 2021-03-09T09:34:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:33:59 smithi146 bash[13136]: debug 2021-03-09T09:33:59.878+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9375 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:34:04.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338007c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:34:04.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338007c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:04.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338007c40 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:04.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338007c40 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f834404c330).stop 2021-03-09T09:34:04.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338007c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:04.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338007c40 unknown :-1 s=CLOSED pgs=3351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:04.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:04.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:04.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338004780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:04.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010660 con 0x7f834c0f4690 2021-03-09T09:34:04.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802dfd0 con 0x7f833800eed0 2021-03-09T09:34:04.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8338062ca0 2021-03-09T09:34:04.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:34:04.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:04.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:34:04.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:04.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.882+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:04.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.886+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:04.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.886+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:04.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.886+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:04.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.886+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:04.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.886+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a910 con 0x7f833800eed0 2021-03-09T09:34:04.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.886+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338004780 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c065040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:04.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c06aed0 con 0x7f833800eed0 2021-03-09T09:34:04.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c06aac0 con 0x7f833800eed0 2021-03-09T09:34:04.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:04.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055310 con 0x7f833800eed0 2021-03-09T09:34:05.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:34:04 smithi146 bash[13136]: debug 2021-03-09T09:34:04.882+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9401 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:34:09.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:34:09.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:09.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338004780 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c065040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:09.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338004780 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c065040).stop 2021-03-09T09:34:09.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338004780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:09.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338004780 unknown :-1 s=CLOSED pgs=3360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:09.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338007c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:09.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:09.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:09.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027780 con 0x7f834c0f4690 2021-03-09T09:34:09.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013720 con 0x7f833802d7b0 2021-03-09T09:34:09.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f4b0 con 0x7f8338062ca0 2021-03-09T09:34:09.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:34:09.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:09.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:34:09.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:09.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.882+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338007c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:09.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.886+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:09.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.886+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:09.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.886+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:09.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.886+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:09.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.886+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:34:09.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.886+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338007c40 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 rx=0x7f833c0346e0 tx=0x7f833c065040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:09.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c019e10 con 0x7f833802d7b0 2021-03-09T09:34:09.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:34:09.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:09.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c069b20 con 0x7f833802d7b0 2021-03-09T09:34:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:34:09 smithi146 bash[13136]: debug 2021-03-09T09:34:09.882+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9426 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:34:14.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338007c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:34:14.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338007c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:14.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338007c40 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 rx=0x7f833c0346e0 tx=0x7f833c065040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:14.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338007c40 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 rx=0x7f833c0346e0 tx=0x7f833c065040).stop 2021-03-09T09:34:14.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338007c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:14.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338007c40 unknown :-1 s=CLOSED pgs=3369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:14.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:14.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:14.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f83380166b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:14.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028f20 con 0x7f834c0f4690 2021-03-09T09:34:14.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018fa0 con 0x7f833800eed0 2021-03-09T09:34:14.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026ad0 con 0x7f8338062ca0 2021-03-09T09:34:14.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:34:14.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:14.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:34:14.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:14.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f83380166b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:14.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:14.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:14.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:14.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:14.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.886+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e4b0 con 0x7f833800eed0 2021-03-09T09:34:14.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f83380166b0 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:14.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c023b90 con 0x7f833800eed0 2021-03-09T09:34:14.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833800eed0 2021-03-09T09:34:14.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:14.894+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05f400 con 0x7f833800eed0 2021-03-09T09:34:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:34:14 smithi146 bash[13136]: debug 2021-03-09T09:34:14.886+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9452 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:34:19.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f83380166b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:34:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f83380166b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f83380166b0 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f83380166b0 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c000bc0).stop 2021-03-09T09:34:19.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f83380166b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:19.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f83380166b0 unknown :-1 s=CLOSED pgs=3376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:19.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:19.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:19.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380084f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:19.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019320 con 0x7f834c0f4690 2021-03-09T09:34:19.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018fa0 con 0x7f833802d7b0 2021-03-09T09:34:19.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020480 con 0x7f8338062ca0 2021-03-09T09:34:19.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:34:19.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:19.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:34:19.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:19.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380084f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:19.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:19.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:19.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:19.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:19.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338063df0 con 0x7f833802d7b0 2021-03-09T09:34:19.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380084f0 secure :-1 s=READY pgs=3385 cs=0 l=1 rev1=1 rx=0x7f833c018b00 tx=0x7f833c000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:19.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01a620 con 0x7f833802d7b0 2021-03-09T09:34:19.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.894+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:34:19.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:19.898+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01a310 con 0x7f833802d7b0 2021-03-09T09:34:20.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:34:19 smithi146 bash[13136]: debug 2021-03-09T09:34:19.886+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9477 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:34:24.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380084f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:34:24.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380084f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:24.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380084f0 secure :-1 s=READY pgs=3385 cs=0 l=1 rev1=1 rx=0x7f833c018b00 tx=0x7f833c000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:24.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380084f0 secure :-1 s=READY pgs=3385 cs=0 l=1 rev1=1 rx=0x7f833c018b00 tx=0x7f833c000bc0).stop 2021-03-09T09:34:24.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380084f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:24.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380084f0 unknown :-1 s=CLOSED pgs=3385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:24.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338002b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:24.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:24.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:24.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065fa0 con 0x7f834c0f4690 2021-03-09T09:34:24.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016130 con 0x7f833800eed0 2021-03-09T09:34:24.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015880 con 0x7f8338062ca0 2021-03-09T09:34:24.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:34:24.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:24.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:34:24.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:24.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338002b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:24.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:24.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:24.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:24.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:24.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833800eed0 2021-03-09T09:34:24.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.894+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338002b00 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 rx=0x7f833c012890 tx=0x7f833c000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:24.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.898+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c069960 con 0x7f833800eed0 2021-03-09T09:34:24.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.898+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833800eed0 2021-03-09T09:34:24.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:24.902+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f833800eed0 2021-03-09T09:34:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:34:24 smithi146 bash[13136]: debug 2021-03-09T09:34:24.890+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9503 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:34:29.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338002b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:34:29.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338002b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:29.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338002b00 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 rx=0x7f833c012890 tx=0x7f833c000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:29.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338002b00 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 rx=0x7f833c012890 tx=0x7f833c000bc0).stop 2021-03-09T09:34:29.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338002b00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:29.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338002b00 unknown :-1 s=CLOSED pgs=3394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:29.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:29.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:29.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338069bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:29.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028f20 con 0x7f834c0f4690 2021-03-09T09:34:29.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018fa0 con 0x7f833802d7b0 2021-03-09T09:34:29.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c9e0 con 0x7f8338062ca0 2021-03-09T09:34:29.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:34:29.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:29.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:34:29.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:29.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338069bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:29.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:29.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:29.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:29.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:29.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b920 con 0x7f833802d7b0 2021-03-09T09:34:29.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338069bd0 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 rx=0x7f833c077cf0 tx=0x7f833c000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:29.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.893+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00f300 con 0x7f833802d7b0 2021-03-09T09:34:29.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.897+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:34:29.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:29.897+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c019be0 con 0x7f833802d7b0 2021-03-09T09:34:30.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:34:29 smithi146 bash[13136]: debug 2021-03-09T09:34:29.890+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9527 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:34:34.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338069bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:34:34.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338069bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:34.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338069bd0 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 rx=0x7f833c077cf0 tx=0x7f833c000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:34.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338069bd0 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 rx=0x7f833c077cf0 tx=0x7f833c000bc0).stop 2021-03-09T09:34:34.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338069bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:34.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338069bd0 unknown :-1 s=CLOSED pgs=3402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:34.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:34.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:34.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338073cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:34.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016130 con 0x7f834c0f4690 2021-03-09T09:34:34.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380131d0 con 0x7f833800eed0 2021-03-09T09:34:34.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018fa0 con 0x7f8338062ca0 2021-03-09T09:34:34.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:34:34.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:34:34.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:34.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:34.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.893+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338073cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:34.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.897+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:34.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.897+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:34.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.897+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:34.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.897+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:34.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.897+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833800eed0 2021-03-09T09:34:34.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.897+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338073cc0 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:34.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.897+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c058910 con 0x7f833800eed0 2021-03-09T09:34:34.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.897+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c058910 con 0x7f833800eed0 2021-03-09T09:34:34.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:34.901+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833800eed0 2021-03-09T09:34:35.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:34:34 smithi146 bash[13136]: debug 2021-03-09T09:34:34.894+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9555 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:34:39.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338073cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:34:39.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338073cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:39.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338073cc0 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:39.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338073cc0 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c000bc0).stop 2021-03-09T09:34:39.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338073cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:39.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338073cc0 unknown :-1 s=CLOSED pgs=3411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:39.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:39.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:39.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380741b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:39.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a510 con 0x7f834c0f4690 2021-03-09T09:34:39.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027780 con 0x7f833802d7b0 2021-03-09T09:34:39.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e890 con 0x7f8338062ca0 2021-03-09T09:34:39.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:34:39.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:39.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:34:39.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:39.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380741b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:39.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:39.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:39.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:39.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:39.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026a80 con 0x7f833802d7b0 2021-03-09T09:34:39.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380741b0 secure :-1 s=READY pgs=3420 cs=0 l=1 rev1=1 rx=0x7f833c042df0 tx=0x7f833c000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:39.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c058910 con 0x7f833802d7b0 2021-03-09T09:34:39.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.897+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c042e20 con 0x7f833802d7b0 2021-03-09T09:34:39.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:39.901+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c058cb0 con 0x7f833802d7b0 2021-03-09T09:34:40.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:34:39 smithi146 bash[13136]: debug 2021-03-09T09:34:39.894+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9578 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:34:44.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380741b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:34:44.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380741b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:44.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380741b0 secure :-1 s=READY pgs=3420 cs=0 l=1 rev1=1 rx=0x7f833c042df0 tx=0x7f833c000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:44.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380741b0 secure :-1 s=READY pgs=3420 cs=0 l=1 rev1=1 rx=0x7f833c042df0 tx=0x7f833c000bc0).stop 2021-03-09T09:34:44.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380741b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:44.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380741b0 unknown :-1 s=CLOSED pgs=3420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:44.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:44.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:44.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:44.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019320 con 0x7f834c0f4690 2021-03-09T09:34:44.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f833800eed0 2021-03-09T09:34:44.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380667f0 con 0x7f8338062ca0 2021-03-09T09:34:44.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:34:44.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:44.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:44.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:34:44.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.897+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:44.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.901+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:44.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.901+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:44.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.901+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:44.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.901+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:44.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.901+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802d620 con 0x7f833800eed0 2021-03-09T09:34:44.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.901+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833806fcb0 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 rx=0x7f833c078d70 tx=0x7f833c063010).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:44.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.901+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c078de0 con 0x7f833800eed0 2021-03-09T09:34:44.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.901+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c078bb0 con 0x7f833800eed0 2021-03-09T09:34:44.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:44.905+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02e350 con 0x7f833800eed0 2021-03-09T09:34:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:34:44 smithi146 bash[13136]: debug 2021-03-09T09:34:44.898+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9605 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:34:49.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833806fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:34:49.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833806fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:49.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833806fcb0 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 rx=0x7f833c078d70 tx=0x7f833c063010).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:49.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833806fcb0 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 rx=0x7f833c078d70 tx=0x7f833c063010).stop 2021-03-09T09:34:49.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833806fcb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:49.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833806fcb0 unknown :-1 s=CLOSED pgs=3429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:49.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:49.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:49.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380721b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:49.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018fa0 con 0x7f834c0f4690 2021-03-09T09:34:49.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016e80 con 0x7f833802d7b0 2021-03-09T09:34:49.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e360 con 0x7f8338062ca0 2021-03-09T09:34:49.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:34:49.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:34:49.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:49.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:49.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380721b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:49.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:49.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:49.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:49.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:49.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.901+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e4b0 con 0x7f833802d7b0 2021-03-09T09:34:49.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.905+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380721b0 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 rx=0x7f833c070d60 tx=0x7f833c000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:49.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.905+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c078de0 con 0x7f833802d7b0 2021-03-09T09:34:49.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.905+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:34:49.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:49.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f833802d7b0 2021-03-09T09:34:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:34:49 smithi146 bash[13136]: debug 2021-03-09T09:34:49.898+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9629 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:34:54.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380721b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:34:54.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380721b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:54.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380721b0 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 rx=0x7f833c070d60 tx=0x7f833c000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:54.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380721b0 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 rx=0x7f833c070d60 tx=0x7f833c000bc0).stop 2021-03-09T09:34:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380721b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380721b0 unknown :-1 s=CLOSED pgs=3439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338012370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028250 con 0x7f834c0f4690 2021-03-09T09:34:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a900 con 0x7f833800eed0 2021-03-09T09:34:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f8338062ca0 2021-03-09T09:34:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:34:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:34:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:54.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338012370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:54.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.901+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.905+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.905+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.905+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.905+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012af0 con 0x7f833800eed0 2021-03-09T09:34:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.905+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338012370 secure :-1 s=READY pgs=3446 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.905+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833800eed0 2021-03-09T09:34:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.905+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033040 con 0x7f833800eed0 2021-03-09T09:34:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:54.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833800eed0 2021-03-09T09:34:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:34:54 smithi146 bash[13136]: debug 2021-03-09T09:34:54.901+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9657 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:34:59.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.901+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338012370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:34:59.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.901+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338012370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:59.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.901+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338012370 secure :-1 s=READY pgs=3446 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:59.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.901+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338012370 secure :-1 s=READY pgs=3446 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834404c330).stop 2021-03-09T09:34:59.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.901+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f8338012370 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:59.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.901+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f8338012370 unknown :-1 s=CLOSED pgs=3446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:59.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.901+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:59.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.901+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:59.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.901+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:59.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.901+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380667f0 con 0x7f834c0f4690 2021-03-09T09:34:59.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.901+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023750 con 0x7f833802d7b0 2021-03-09T09:34:59.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.901+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018fa0 con 0x7f8338062ca0 2021-03-09T09:34:59.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.901+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:34:59.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.901+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:59.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.901+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:34:59.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.901+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:59.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.905+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:59.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.905+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:59.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.905+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:59.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.905+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:59.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.905+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:59.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.905+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014750 con 0x7f833802d7b0 2021-03-09T09:34:59.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.905+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071cc0 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 rx=0x7f833c036fe0 tx=0x7f833c000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:59.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c040970 con 0x7f833802d7b0 2021-03-09T09:34:59.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:34:59.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:34:59.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018b00 con 0x7f833802d7b0 2021-03-09T09:35:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:34:59 smithi146 bash[13136]: debug 2021-03-09T09:34:59.901+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9680 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:35:04.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:35:04.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:04.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071cc0 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 rx=0x7f833c036fe0 tx=0x7f833c000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:04.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071cc0 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 rx=0x7f833c036fe0 tx=0x7f833c000bc0).stop 2021-03-09T09:35:04.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:04.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071cc0 unknown :-1 s=CLOSED pgs=3454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:04.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:04.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:04.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833802d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:04.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026ad0 con 0x7f834c0f4690 2021-03-09T09:35:04.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a900 con 0x7f833800eed0 2021-03-09T09:35:04.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fb60 con 0x7f8338062ca0 2021-03-09T09:35:04.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:35:04.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:04.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:35:04.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:04.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833802d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:04.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:04.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:04.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:04.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:04.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b920 con 0x7f833800eed0 2021-03-09T09:35:04.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.909+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833802d050 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:04.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.913+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c036340 con 0x7f833800eed0 2021-03-09T09:35:04.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.913+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833800eed0 2021-03-09T09:35:04.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:04.917+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0230a0 con 0x7f833800eed0 2021-03-09T09:35:05.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:35:04 smithi146 bash[13136]: debug 2021-03-09T09:35:04.905+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9707 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:35:09.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.905+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833802d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:35:09.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.905+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833802d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:09.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.905+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833802d050 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:09.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.905+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833802d050 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c000bc0).stop 2021-03-09T09:35:09.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 msgr2=0x7f833802d050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:09.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800eed0 0x7f833802d050 unknown :-1 s=CLOSED pgs=3465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:09.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:09.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:09.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:09.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380131d0 con 0x7f834c0f4690 2021-03-09T09:35:09.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f833802d7b0 2021-03-09T09:35:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068d50 con 0x7f8338062ca0 2021-03-09T09:35:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:35:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:35:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:09.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:09.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e5b0 con 0x7f833802d7b0 2021-03-09T09:35:09.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071cc0 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:09.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833802d7b0 2021-03-09T09:35:09.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.913+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833802d7b0 2021-03-09T09:35:09.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:09.913+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402acd0 con 0x7f833802d7b0 2021-03-09T09:35:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:35:09 smithi146 bash[13136]: debug 2021-03-09T09:35:09.909+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9731 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:35:14.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:35:14.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:14.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071cc0 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:14.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071cc0 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834404c330).stop 2021-03-09T09:35:14.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:14.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071cc0 unknown :-1 s=CLOSED pgs=3472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:14.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338066f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:14.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:14.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:14.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018fa0 con 0x7f834c0f4690 2021-03-09T09:35:14.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064990 con 0x7f8338066b90 2021-03-09T09:35:14.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f8338062ca0 2021-03-09T09:35:14.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:35:14.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:14.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338066f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:14.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:35:14.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:14.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:14.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:14.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:14.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:14.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f8338066b90 2021-03-09T09:35:14.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338066f70 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 rx=0x7f833c0197a0 tx=0x7f833c04d2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:14.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f8338066b90 2021-03-09T09:35:14.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.913+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8338066b90 2021-03-09T09:35:14.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:14.917+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054310 con 0x7f8338066b90 2021-03-09T09:35:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:35:14 smithi146 bash[13136]: debug 2021-03-09T09:35:14.909+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9758 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:35:19.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338066f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:35:19.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338066f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:19.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338066f70 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 rx=0x7f833c0197a0 tx=0x7f833c04d2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:19.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338066f70 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 rx=0x7f833c0197a0 tx=0x7f833c04d2e0).stop 2021-03-09T09:35:19.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338066f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:19.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338066f70 unknown :-1 s=CLOSED pgs=3480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:19.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:19.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:19.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:19.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068d50 con 0x7f834c0f4690 2021-03-09T09:35:19.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f833802d7b0 2021-03-09T09:35:19.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380181a0 con 0x7f8338062ca0 2021-03-09T09:35:19.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:35:19.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:19.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:19.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:35:19.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.912+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:19.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:19.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:19.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:19.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:19.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e4b0 con 0x7f833802d7b0 2021-03-09T09:35:19.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071ca0 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 rx=0x7f833c0403b0 tx=0x7f833c04d2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:19.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.916+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c058690 con 0x7f833802d7b0 2021-03-09T09:35:19.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.916+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c058690 con 0x7f833802d7b0 2021-03-09T09:35:19.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:19.920+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01a270 con 0x7f833802d7b0 2021-03-09T09:35:20.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:35:19 smithi146 bash[13136]: debug 2021-03-09T09:35:19.913+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9782 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:35:24.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:35:24.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:24.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071ca0 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 rx=0x7f833c0403b0 tx=0x7f833c04d2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:24.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071ca0 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 rx=0x7f833c0403b0 tx=0x7f833c04d2e0).stop 2021-03-09T09:35:24.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071ca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:24.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071ca0 unknown :-1 s=CLOSED pgs=3492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:24.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:24.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:24.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:24.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021610 con 0x7f834c0f4690 2021-03-09T09:35:24.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e890 con 0x7f8338066b90 2021-03-09T09:35:24.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067640 con 0x7f8338062ca0 2021-03-09T09:35:24.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:35:24.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:24.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:35:24.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:24.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:24.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:24.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:24.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:24.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:24.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f8338066b90 2021-03-09T09:35:24.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338006e50 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c04d2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:24.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c040970 con 0x7f8338066b90 2021-03-09T09:35:24.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.916+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c040ad0 con 0x7f8338066b90 2021-03-09T09:35:24.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:24.920+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0252e0 con 0x7f8338066b90 2021-03-09T09:35:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:35:24 smithi146 bash[13136]: debug 2021-03-09T09:35:24.913+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9809 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:35:29.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:35:29.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:29.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338006e50 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c04d2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:29.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338006e50 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c04d2e0).stop 2021-03-09T09:35:29.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:29.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338006e50 unknown :-1 s=CLOSED pgs=3499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:29.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:29.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:29.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380706d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:29.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d750 con 0x7f834c0f4690 2021-03-09T09:35:29.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018fa0 con 0x7f833802d7b0 2021-03-09T09:35:29.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068d50 con 0x7f8338062ca0 2021-03-09T09:35:29.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:35:29.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:29.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:35:29.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:29.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:29.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:29.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:29.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:29.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:29.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a890 con 0x7f833802d7b0 2021-03-09T09:35:29.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.920+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380706d0 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 rx=0x7f833c04d4a0 tx=0x7f833c04d2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:29.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.924+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c036340 con 0x7f833802d7b0 2021-03-09T09:35:29.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.924+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01b290 con 0x7f833802d7b0 2021-03-09T09:35:29.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:29.928+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01a270 con 0x7f833802d7b0 2021-03-09T09:35:30.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:35:29 smithi146 bash[13136]: debug 2021-03-09T09:35:29.917+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9833 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:35:34.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:35:34.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:34.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380706d0 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 rx=0x7f833c04d4a0 tx=0x7f833c04d2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:34.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380706d0 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 rx=0x7f833c04d4a0 tx=0x7f833c04d2e0).stop 2021-03-09T09:35:34.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.916+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380706d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:34.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.916+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380706d0 unknown :-1 s=CLOSED pgs=3509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:34.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.916+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:34.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.916+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338007810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:34.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.916+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:34.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.916+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061550 con 0x7f834c0f4690 2021-03-09T09:35:34.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.916+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380098d0 con 0x7f8338066b90 2021-03-09T09:35:34.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.916+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067670 con 0x7f8338062ca0 2021-03-09T09:35:34.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:35:34.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:34.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.916+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:35:34.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.916+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:34.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.920+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338007810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:34.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.920+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:34.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.920+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:34.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.920+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:34.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.920+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:34.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.920+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338009a10 con 0x7f8338066b90 2021-03-09T09:35:34.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.920+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338007810 secure :-1 s=READY pgs=3517 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:34.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.924+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338066b90 2021-03-09T09:35:34.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.924+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338066b90 2021-03-09T09:35:34.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:34.928+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402acd0 con 0x7f8338066b90 2021-03-09T09:35:35.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:35:34 smithi146 bash[13136]: debug 2021-03-09T09:35:34.917+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9861 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:35:39.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338007810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:35:39.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338007810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:39.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338007810 secure :-1 s=READY pgs=3517 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:39.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338007810 secure :-1 s=READY pgs=3517 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404c330).stop 2021-03-09T09:35:39.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338007810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:39.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338007810 unknown :-1 s=CLOSED pgs=3517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:39.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:39.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338004780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:39.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:39.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018c10 con 0x7f834c0f4690 2021-03-09T09:35:39.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f833802d7b0 2021-03-09T09:35:39.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018fa0 con 0x7f8338062ca0 2021-03-09T09:35:39.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:35:39.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:39.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:35:39.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:39.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.920+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:39.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.924+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:39.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.924+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:39.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.924+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:39.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.924+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:39.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.924+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833805eb00 con 0x7f833802d7b0 2021-03-09T09:35:39.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.924+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338004780 secure :-1 s=READY pgs=3526 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:39.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.924+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344072b60 con 0x7f833802d7b0 2021-03-09T09:35:39.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.924+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072980 con 0x7f833802d7b0 2021-03-09T09:35:39.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:39.932+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344022a80 con 0x7f833802d7b0 2021-03-09T09:35:40.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:35:39 smithi146 bash[13136]: debug 2021-03-09T09:35:39.921+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9884 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:35:44.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.920+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:35:44.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.920+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:44.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.920+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338004780 secure :-1 s=READY pgs=3526 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:44.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.920+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338004780 secure :-1 s=READY pgs=3526 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834404c330).stop 2021-03-09T09:35:44.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.920+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338004780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:44.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.920+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338004780 unknown :-1 s=CLOSED pgs=3526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:44.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.920+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:44.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.920+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:44.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.920+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:44.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.920+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020680 con 0x7f834c0f4690 2021-03-09T09:35:44.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.920+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380669a0 con 0x7f8338066b90 2021-03-09T09:35:44.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.920+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801eff0 con 0x7f8338062ca0 2021-03-09T09:35:44.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.920+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:35:44.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.920+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:44.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.920+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:35:44.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.920+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:44.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.924+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:44.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.924+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:44.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.924+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:44.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.924+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:44.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.924+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.924+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014020 con 0x7f8338066b90 2021-03-09T09:35:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.924+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338070db0 secure :-1 s=READY pgs=3533 cs=0 l=1 rev1=1 rx=0x7f8344054cd0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.928+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338066b90 2021-03-09T09:35:44.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.928+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072b60 con 0x7f8338066b90 2021-03-09T09:35:44.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:44.928+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402bde0 con 0x7f8338066b90 2021-03-09T09:35:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:35:44 smithi146 bash[13136]: debug 2021-03-09T09:35:44.921+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9910 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:35:49.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:35:49.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:49.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338070db0 secure :-1 s=READY pgs=3533 cs=0 l=1 rev1=1 rx=0x7f8344054cd0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:49.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338070db0 secure :-1 s=READY pgs=3533 cs=0 l=1 rev1=1 rx=0x7f8344054cd0 tx=0x7f834404c330).stop 2021-03-09T09:35:49.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338070db0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:49.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338070db0 unknown :-1 s=CLOSED pgs=3533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:49.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:49.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:49.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:49.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023800 con 0x7f834c0f4690 2021-03-09T09:35:49.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007230 con 0x7f833802d7b0 2021-03-09T09:35:49.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380251b0 con 0x7f8338062ca0 2021-03-09T09:35:49.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:35:49.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:49.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:35:49.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:49.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:49.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:49.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:49.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:49.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.924+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:49.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.928+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:35:49.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.928+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801dd10 secure :-1 s=READY pgs=3541 cs=0 l=1 rev1=1 rx=0x7f833c040c10 tx=0x7f833c04d2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:49.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.928+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833802d7b0 2021-03-09T09:35:49.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.928+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:35:49.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:49.932+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f833802d7b0 2021-03-09T09:35:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:35:49 smithi146 bash[13136]: debug 2021-03-09T09:35:49.925+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9935 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:35:54.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:35:54.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:54.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801dd10 secure :-1 s=READY pgs=3541 cs=0 l=1 rev1=1 rx=0x7f833c040c10 tx=0x7f833c04d2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:54.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801dd10 secure :-1 s=READY pgs=3541 cs=0 l=1 rev1=1 rx=0x7f833c040c10 tx=0x7f833c04d2e0).stop 2021-03-09T09:35:54.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833801dd10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:54.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833801dd10 unknown :-1 s=CLOSED pgs=3541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:54.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338027990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:54.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:54.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:54.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064b90 con 0x7f834c0f4690 2021-03-09T09:35:54.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018fa0 con 0x7f8338066b90 2021-03-09T09:35:54.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8338062ca0 2021-03-09T09:35:54.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:35:54.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:54.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338027990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:54.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:35:54.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:54.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:54.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:54.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:54.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:54.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a290 con 0x7f8338066b90 2021-03-09T09:35:54.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338027990 secure :-1 s=READY pgs=3550 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c04d2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:54.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c040240 con 0x7f8338066b90 2021-03-09T09:35:54.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.928+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c054820 con 0x7f8338066b90 2021-03-09T09:35:54.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:54.932+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c078bf0 con 0x7f8338066b90 2021-03-09T09:35:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:35:54 smithi146 bash[13136]: debug 2021-03-09T09:35:54.925+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9961 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:35:59.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338027990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:35:59.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338027990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:59.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338027990 secure :-1 s=READY pgs=3550 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c04d2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:59.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338027990 secure :-1 s=READY pgs=3550 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c04d2e0).stop 2021-03-09T09:35:59.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338027990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:59.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338027990 unknown :-1 s=CLOSED pgs=3550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:59.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:59.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:59.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:59.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018c10 con 0x7f834c0f4690 2021-03-09T09:35:59.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f833802d7b0 2021-03-09T09:35:59.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018fa0 con 0x7f8338062ca0 2021-03-09T09:35:59.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:35:59.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:59.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:35:59.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:59.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:59.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:59.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:59.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:59.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:59.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338063df0 con 0x7f833802d7b0 2021-03-09T09:35:59.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 rx=0x7f8344027ac0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:59.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f833802d7b0 2021-03-09T09:35:59.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.932+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833802d7b0 2021-03-09T09:35:59.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:35:59.936+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344054b90 con 0x7f833802d7b0 2021-03-09T09:36:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:35:59 smithi146 bash[13136]: debug 2021-03-09T09:35:59.929+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 9986 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:36:04.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338072dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:36:04.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338072dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:04.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 rx=0x7f8344027ac0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:04.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 rx=0x7f8344027ac0 tx=0x7f834404c330).stop 2021-03-09T09:36:04.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338072dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:04.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 unknown :-1 s=CLOSED pgs=3559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:04.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:04.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:04.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338028550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:04.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005570 con 0x7f834c0f4690 2021-03-09T09:36:04.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e360 con 0x7f8338066b90 2021-03-09T09:36:04.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d510 con 0x7f8338062ca0 2021-03-09T09:36:04.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:36:04.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:04.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:36:04.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:04.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338028550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:04.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:04.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.931+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380136e0 con 0x7f8338066b90 2021-03-09T09:36:04.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.935+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338028550 secure :-1 s=READY pgs=3569 cs=0 l=1 rev1=1 rx=0x7f833c0235d0 tx=0x7f833c04d2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:04.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f8338066b90 2021-03-09T09:36:04.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338066b90 2021-03-09T09:36:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:04.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338066b90 2021-03-09T09:36:05.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:36:04 smithi146 bash[13136]: debug 2021-03-09T09:36:04.932+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10012 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:36:09.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338028550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:36:09.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338028550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:09.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338028550 secure :-1 s=READY pgs=3569 cs=0 l=1 rev1=1 rx=0x7f833c0235d0 tx=0x7f833c04d2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:09.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338028550 secure :-1 s=READY pgs=3569 cs=0 l=1 rev1=1 rx=0x7f833c0235d0 tx=0x7f833c04d2e0).stop 2021-03-09T09:36:09.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338028550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:09.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338028550 unknown :-1 s=CLOSED pgs=3569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:09.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:09.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:09.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:09.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:36:09.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:09.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014900 con 0x7f834c0f4690 2021-03-09T09:36:09.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023800 con 0x7f833802d7b0 2021-03-09T09:36:09.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007230 con 0x7f8338062ca0 2021-03-09T09:36:09.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:36:09.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:09.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:09.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:09.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:09.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:09.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:09.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.935+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024c50 con 0x7f833802d7b0 2021-03-09T09:36:09.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.939+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 secure :-1 s=READY pgs=3576 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:09.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833802d7b0 2021-03-09T09:36:09.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033040 con 0x7f833802d7b0 2021-03-09T09:36:09.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:09.943+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833802d7b0 2021-03-09T09:36:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:36:09 smithi146 bash[13136]: debug 2021-03-09T09:36:09.932+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10036 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:36:14.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338072dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:36:14.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338072dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:14.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 secure :-1 s=READY pgs=3576 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:14.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 secure :-1 s=READY pgs=3576 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834404c330).stop 2021-03-09T09:36:14.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338072dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:14.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 unknown :-1 s=CLOSED pgs=3576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:14.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:14.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:14.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:14.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380251b0 con 0x7f834c0f4690 2021-03-09T09:36:14.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018fa0 con 0x7f8338066b90 2021-03-09T09:36:14.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069760 con 0x7f8338062ca0 2021-03-09T09:36:14.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:36:14.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:14.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:36:14.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:14.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.935+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:14.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.939+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:14.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.939+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:14.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.939+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:14.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.939+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:14.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.939+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068d00 con 0x7f8338066b90 2021-03-09T09:36:14.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.939+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338021080 secure :-1 s=READY pgs=3584 cs=0 l=1 rev1=1 rx=0x7f834400ff50 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:14.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f8338066b90 2021-03-09T09:36:14.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033040 con 0x7f8338066b90 2021-03-09T09:36:14.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:14.943+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344022800 con 0x7f8338066b90 2021-03-09T09:36:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:36:14 smithi146 bash[13136]: debug 2021-03-09T09:36:14.932+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10062 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:36:19.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:36:19.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:19.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338021080 secure :-1 s=READY pgs=3584 cs=0 l=1 rev1=1 rx=0x7f834400ff50 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:19.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338021080 secure :-1 s=READY pgs=3584 cs=0 l=1 rev1=1 rx=0x7f834400ff50 tx=0x7f834404c330).stop 2021-03-09T09:36:19.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:19.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338021080 unknown :-1 s=CLOSED pgs=3584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:19.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:19.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:19.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:19.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e360 con 0x7f834c0f4690 2021-03-09T09:36:19.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b970 con 0x7f833802d7b0 2021-03-09T09:36:19.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8338062ca0 2021-03-09T09:36:19.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:36:19.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:19.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:36:19.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:19.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:19.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:19.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:19.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:19.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:19.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:36:19.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.939+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 secure :-1 s=READY pgs=3593 cs=0 l=1 rev1=1 rx=0x7f83440631f0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:19.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.943+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344071740 con 0x7f833802d7b0 2021-03-09T09:36:19.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.943+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033040 con 0x7f833802d7b0 2021-03-09T09:36:19.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:19.947+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f833802d7b0 2021-03-09T09:36:20.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:36:19 smithi146 bash[13136]: debug 2021-03-09T09:36:19.936+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10088 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:36:24.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.939+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338072dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:36:24.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.939+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338072dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:24.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.939+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 secure :-1 s=READY pgs=3593 cs=0 l=1 rev1=1 rx=0x7f83440631f0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:24.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.939+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 secure :-1 s=READY pgs=3593 cs=0 l=1 rev1=1 rx=0x7f83440631f0 tx=0x7f834404c330).stop 2021-03-09T09:36:24.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338072dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:24.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.939+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338072dc0 unknown :-1 s=CLOSED pgs=3593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:24.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.939+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:24.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.939+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:24.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.939+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338004780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:24.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f834c0f4690 2021-03-09T09:36:24.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c7d0 con 0x7f8338066b90 2021-03-09T09:36:24.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010050 con 0x7f8338062ca0 2021-03-09T09:36:24.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:36:24.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:24.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:36:24.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:24.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:24.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 msgr2=0x7f8338063240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:24.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8338062ca0 0x7f8338063240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:24.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 msgr2=0x7f8338004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:36:24.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338066b90 0x7f8338004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:24.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 msgr2=0x7f8338060780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:24.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834c0f4690 0x7f8338060780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:24.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8324005850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:24.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8324008290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:24.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f832400ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:24.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f8324003400 2021-03-09T09:36:24.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c7d0 con 0x7f8324008870 2021-03-09T09:36:24.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010050 con 0x7f8324005e00 2021-03-09T09:36:24.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8324005850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:36:24.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8324005850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:24.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8324008290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:36:24.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.943+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8324008290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:24.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.947+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f832400ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:24.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.947+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8324008290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:24.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.947+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8324008290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:24.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.947+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8324005850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:24.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.947+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8324005850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:24.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.947+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f8324008870 2021-03-09T09:36:24.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.947+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f832400ad00 secure :-1 s=READY pgs=3605 cs=0 l=1 rev1=1 rx=0x7f833c063050 tx=0x7f833c04d2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:24.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0094a0 con 0x7f8324008870 2021-03-09T09:36:24.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8324008870 2021-03-09T09:36:24.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:24.955+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8324008870 2021-03-09T09:36:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:36:24 smithi146 bash[13136]: debug 2021-03-09T09:36:24.940+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10113 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:36:29.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f832400ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:36:29.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f832400ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:29.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f832400ad00 secure :-1 s=READY pgs=3605 cs=0 l=1 rev1=1 rx=0x7f833c063050 tx=0x7f833c04d2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:29.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f832400ad00 secure :-1 s=READY pgs=3605 cs=0 l=1 rev1=1 rx=0x7f833c063050 tx=0x7f833c04d2e0).stop 2021-03-09T09:36:29.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f832400ad00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:29.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f832400ad00 unknown :-1 s=CLOSED pgs=3605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:29.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:29.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:29.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:29.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800bf00 con 0x7f8324003400 2021-03-09T09:36:29.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067670 con 0x7f833802d7b0 2021-03-09T09:36:29.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:36:29.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:36:29.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:29.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:36:29.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:29.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.943+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:29.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.947+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:29.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.947+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:29.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.947+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:29.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.947+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:29.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.947+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068d00 con 0x7f833802d7b0 2021-03-09T09:36:29.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.947+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3610 cs=0 l=1 rev1=1 rx=0x7f833c026040 tx=0x7f833c04d2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:29.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.947+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c040ae0 con 0x7f833802d7b0 2021-03-09T09:36:29.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.947+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:36:29.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:29.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c019590 con 0x7f833802d7b0 2021-03-09T09:36:30.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:36:29 smithi146 bash[13136]: debug 2021-03-09T09:36:29.940+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10139 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:36:34.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:36:34.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:34.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3610 cs=0 l=1 rev1=1 rx=0x7f833c026040 tx=0x7f833c04d2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:34.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3610 cs=0 l=1 rev1=1 rx=0x7f833c026040 tx=0x7f833c04d2e0).stop 2021-03-09T09:36:34.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:34.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:34.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:34.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:34.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:34.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010050 con 0x7f8324003400 2021-03-09T09:36:34.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800bf00 con 0x7f8324008870 2021-03-09T09:36:34.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b2d0 con 0x7f8324005e00 2021-03-09T09:36:34.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:36:34.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:36:34.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:34.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:34.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338060780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:34.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:34.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:34.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:34.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:34.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f8324008870 2021-03-09T09:36:34.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.947+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338060780 secure :-1 s=READY pgs=3619 cs=0 l=1 rev1=1 rx=0x7f833c058010 tx=0x7f833c04d2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:34.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05fbd0 con 0x7f8324008870 2021-03-09T09:36:34.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05f9f0 con 0x7f8324008870 2021-03-09T09:36:34.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:34.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c019590 con 0x7f8324008870 2021-03-09T09:36:35.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:36:34 smithi146 bash[13136]: debug 2021-03-09T09:36:34.944+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10165 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:36:39.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.947+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338060780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:36:39.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.947+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338060780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:39.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.947+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338060780 secure :-1 s=READY pgs=3619 cs=0 l=1 rev1=1 rx=0x7f833c058010 tx=0x7f833c04d2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:39.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.947+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338060780 secure :-1 s=READY pgs=3619 cs=0 l=1 rev1=1 rx=0x7f833c058010 tx=0x7f833c04d2e0).stop 2021-03-09T09:36:39.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.947+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338060780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:39.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.947+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338060780 unknown :-1 s=CLOSED pgs=3619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:39.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.947+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:39.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.947+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:39.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.947+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:36:39.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.947+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:39.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.951+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:39.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.951+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:39.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806df70 con 0x7f8324003400 2021-03-09T09:36:39.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023800 con 0x7f833802d7b0 2021-03-09T09:36:39.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.951+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:36:39.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.951+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:39.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007230 con 0x7f8324005e00 2021-03-09T09:36:39.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.951+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:39.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.951+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:39.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.951+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:39.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.951+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:39.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.951+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800e310 con 0x7f833802d7b0 2021-03-09T09:36:39.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.951+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3628 cs=0 l=1 rev1=1 rx=0x7f833c037040 tx=0x7f833c04d2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:39.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f833802d7b0 2021-03-09T09:36:39.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05f9f0 con 0x7f833802d7b0 2021-03-09T09:36:39.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:39.955+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054310 con 0x7f833802d7b0 2021-03-09T09:36:40.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:36:39 smithi146 bash[13136]: debug 2021-03-09T09:36:39.948+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10191 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:36:44.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:36:44.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:44.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3628 cs=0 l=1 rev1=1 rx=0x7f833c037040 tx=0x7f833c04d2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:44.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3628 cs=0 l=1 rev1=1 rx=0x7f833c037040 tx=0x7f833c04d2e0).stop 2021-03-09T09:36:44.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:44.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:44.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338061060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:44.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:44.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:44.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010050 con 0x7f8324003400 2021-03-09T09:36:44.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380153f0 con 0x7f8324008870 2021-03-09T09:36:44.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e360 con 0x7f8324005e00 2021-03-09T09:36:44.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:36:44.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:44.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:36:44.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:44.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338061060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:44.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:44.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:44.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:44.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:44.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.951+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd50 con 0x7f8324008870 2021-03-09T09:36:44.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.955+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338061060 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 rx=0x7f833c060000 tx=0x7f833c04d2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:44.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.955+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c060260 con 0x7f8324008870 2021-03-09T09:36:44.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.955+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c018670 con 0x7f8324008870 2021-03-09T09:36:44.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:44.959+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054310 con 0x7f8324008870 2021-03-09T09:36:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:36:44 smithi146 bash[13136]: debug 2021-03-09T09:36:44.948+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10215 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:36:49.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338061060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:36:49.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338061060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:49.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338061060 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 rx=0x7f833c060000 tx=0x7f833c04d2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:49.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338061060 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 rx=0x7f833c060000 tx=0x7f833c04d2e0).stop 2021-03-09T09:36:49.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338061060 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:49.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338061060 unknown :-1 s=CLOSED pgs=3636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:49.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:49.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:49.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:49.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800bf00 con 0x7f8324003400 2021-03-09T09:36:49.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b2d0 con 0x7f833802d7b0 2021-03-09T09:36:49.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018f70 con 0x7f8324005e00 2021-03-09T09:36:49.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:36:49.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:49.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:36:49.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:49.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.951+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:49.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.955+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:49.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.955+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:49.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.955+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:49.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.955+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:49.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.955+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:36:49.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.955+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3645 cs=0 l=1 rev1=1 rx=0x7f8338017f00 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:49.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.955+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344071740 con 0x7f833802d7b0 2021-03-09T09:36:49.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.955+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833802d7b0 2021-03-09T09:36:49.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:49.959+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833802d7b0 2021-03-09T09:36:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:36:49 smithi146 bash[13136]: debug 2021-03-09T09:36:49.952+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10241 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:36:54.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:36:54.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:54.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3645 cs=0 l=1 rev1=1 rx=0x7f8338017f00 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:54.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3645 cs=0 l=1 rev1=1 rx=0x7f8338017f00 tx=0x7f834404c330).stop 2021-03-09T09:36:54.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:54.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:54.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:54.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:54.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:54.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023800 con 0x7f8324003400 2021-03-09T09:36:54.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c7d0 con 0x7f8324008870 2021-03-09T09:36:54.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010050 con 0x7f8324005e00 2021-03-09T09:36:54.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:36:54.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:54.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:36:54.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:54.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.954+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:54.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.958+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:54.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.958+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:54.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.958+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:54.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.958+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:54.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.958+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f8324008870 2021-03-09T09:36:54.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.958+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016620 secure :-1 s=READY pgs=3658 cs=0 l=1 rev1=1 rx=0x7f833c043050 tx=0x7f833c04d2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:54.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00f300 con 0x7f8324008870 2021-03-09T09:36:54.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8324008870 2021-03-09T09:36:54.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:54.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0572b0 con 0x7f8324008870 2021-03-09T09:36:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:36:54 smithi146 bash[13136]: debug 2021-03-09T09:36:54.952+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10266 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:36:59.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.954+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338016620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:36:59.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.954+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338016620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:59.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.954+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016620 secure :-1 s=READY pgs=3658 cs=0 l=1 rev1=1 rx=0x7f833c043050 tx=0x7f833c04d2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:59.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.954+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016620 secure :-1 s=READY pgs=3658 cs=0 l=1 rev1=1 rx=0x7f833c043050 tx=0x7f833c04d2e0).stop 2021-03-09T09:36:59.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338016620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:59.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.954+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016620 unknown :-1 s=CLOSED pgs=3658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:59.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.954+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:59.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.954+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:59.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.954+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:59.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801af60 con 0x7f8324003400 2021-03-09T09:36:59.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b2d0 con 0x7f833802d7b0 2021-03-09T09:36:59.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018f70 con 0x7f8324005e00 2021-03-09T09:36:59.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.954+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:36:59.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.954+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:59.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.958+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:36:59.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.958+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:59.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.958+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:59.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.958+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:59.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.958+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:59.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.958+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:59.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.958+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:59.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.958+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:36:59.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.958+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3662 cs=0 l=1 rev1=1 rx=0x7f8344079f90 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:59.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029030 con 0x7f833802d7b0 2021-03-09T09:36:59.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833802d7b0 2021-03-09T09:36:59.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:36:59.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344054b60 con 0x7f833802d7b0 2021-03-09T09:37:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:36:59 smithi146 bash[13136]: debug 2021-03-09T09:36:59.956+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10293 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:37:04.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:37:04.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:04.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3662 cs=0 l=1 rev1=1 rx=0x7f8344079f90 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:04.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3662 cs=0 l=1 rev1=1 rx=0x7f8344079f90 tx=0x7f834404c330).stop 2021-03-09T09:37:04.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:04.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:04.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:04.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:04.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:04.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010050 con 0x7f8324003400 2021-03-09T09:37:04.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800bf00 con 0x7f8324008870 2021-03-09T09:37:04.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e360 con 0x7f8324005e00 2021-03-09T09:37:04.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:37:04.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:04.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:37:04.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:04.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.958+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:04.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.962+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:04.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.962+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:04.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.962+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:04.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.962+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:04.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.962+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068790 con 0x7f8324008870 2021-03-09T09:37:04.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.962+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069af0 secure :-1 s=READY pgs=3671 cs=0 l=1 rev1=1 rx=0x7f834404bae0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:04.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f8324008870 2021-03-09T09:37:04.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344078190 con 0x7f8324008870 2021-03-09T09:37:04.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:04.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834401f5b0 con 0x7f8324008870 2021-03-09T09:37:05.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:37:04 smithi146 bash[13136]: debug 2021-03-09T09:37:04.960+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10317 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:37:09.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.958+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:37:09.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.958+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:09.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.958+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069af0 secure :-1 s=READY pgs=3671 cs=0 l=1 rev1=1 rx=0x7f834404bae0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:09.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.958+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069af0 secure :-1 s=READY pgs=3671 cs=0 l=1 rev1=1 rx=0x7f834404bae0 tx=0x7f834404c330).stop 2021-03-09T09:37:09.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:09.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.958+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069af0 unknown :-1 s=CLOSED pgs=3671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:09.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:09.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018f70 con 0x7f8324003400 2021-03-09T09:37:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026960 con 0x7f833802d7b0 2021-03-09T09:37:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d630 con 0x7f8324005e00 2021-03-09T09:37:09.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:37:09.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:10.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:37:10.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:10.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:10.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:10.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:10.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:10.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:10.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338009a10 con 0x7f833802d7b0 2021-03-09T09:37:10.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3680 cs=0 l=1 rev1=1 rx=0x7f833c071720 tx=0x7f833c04d2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:10.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f833802d7b0 2021-03-09T09:37:10.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c055310 con 0x7f833802d7b0 2021-03-09T09:37:10.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:09.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c040970 con 0x7f833802d7b0 2021-03-09T09:37:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:37:09 smithi146 bash[13136]: debug 2021-03-09T09:37:09.964+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10343 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:37:14.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:37:14.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:14.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3680 cs=0 l=1 rev1=1 rx=0x7f833c071720 tx=0x7f833c04d2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:14.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3680 cs=0 l=1 rev1=1 rx=0x7f833c071720 tx=0x7f833c04d2e0).stop 2021-03-09T09:37:14.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:14.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:14.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:14.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:14.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833806a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010050 con 0x7f8324003400 2021-03-09T09:37:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028d40 con 0x7f8324008870 2021-03-09T09:37:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003060 con 0x7f8324005e00 2021-03-09T09:37:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:37:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:14.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:37:14.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:14.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833806a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:14.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:14.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:14.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:14.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:14.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f8324008870 2021-03-09T09:37:14.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833806a330 secure :-1 s=READY pgs=3688 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c055480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:14.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c011870 con 0x7f8324008870 2021-03-09T09:37:14.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0119d0 con 0x7f8324008870 2021-03-09T09:37:14.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:14.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c011c60 con 0x7f8324008870 2021-03-09T09:37:15.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:37:14 smithi146 bash[13136]: debug 2021-03-09T09:37:14.964+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10368 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:37:19.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.962+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833806a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:37:19.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.962+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833806a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:19.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833806a330 secure :-1 s=READY pgs=3688 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c055480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:19.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833806a330 secure :-1 s=READY pgs=3688 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c055480).stop 2021-03-09T09:37:19.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833806a330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:19.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833806a330 unknown :-1 s=CLOSED pgs=3688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:19.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:19.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:19.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:19.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010050 con 0x7f8324003400 2021-03-09T09:37:19.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e360 con 0x7f833802d7b0 2021-03-09T09:37:19.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067470 con 0x7f8324005e00 2021-03-09T09:37:19.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:37:19.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:37:19.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:19.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:19.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:19.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:19.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:19.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:19.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:19.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068790 con 0x7f833802d7b0 2021-03-09T09:37:19.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3698 cs=0 l=1 rev1=1 rx=0x7f833c025d20 tx=0x7f833c055480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:19.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c025750 con 0x7f833802d7b0 2021-03-09T09:37:19.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0353a0 con 0x7f833802d7b0 2021-03-09T09:37:19.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:19.974+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054310 con 0x7f833802d7b0 2021-03-09T09:37:20.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:37:19 smithi146 bash[13136]: debug 2021-03-09T09:37:19.967+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10394 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:37:24.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:37:24.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:24.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3698 cs=0 l=1 rev1=1 rx=0x7f833c025d20 tx=0x7f833c055480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:24.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3698 cs=0 l=1 rev1=1 rx=0x7f833c025d20 tx=0x7f833c055480).stop 2021-03-09T09:37:24.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:24.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:24.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:24.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:24.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:24.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023800 con 0x7f8324003400 2021-03-09T09:37:24.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010050 con 0x7f8324008870 2021-03-09T09:37:24.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017b60 con 0x7f8324005e00 2021-03-09T09:37:24.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:37:24.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:24.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:37:24.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:24.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:24.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:24.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:24.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:24.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:24.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338009a10 con 0x7f8324008870 2021-03-09T09:37:24.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802c7b0 secure :-1 s=READY pgs=3708 cs=0 l=1 rev1=1 rx=0x7f833c053cb0 tx=0x7f833c055480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:24.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c025750 con 0x7f8324008870 2021-03-09T09:37:24.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c078220 con 0x7f8324008870 2021-03-09T09:37:24.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:24.974+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c040970 con 0x7f8324008870 2021-03-09T09:37:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:37:24 smithi146 bash[13136]: debug 2021-03-09T09:37:24.967+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10420 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:37:29.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833802c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:37:29.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833802c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:29.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802c7b0 secure :-1 s=READY pgs=3708 cs=0 l=1 rev1=1 rx=0x7f833c053cb0 tx=0x7f833c055480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:29.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802c7b0 secure :-1 s=READY pgs=3708 cs=0 l=1 rev1=1 rx=0x7f833c053cb0 tx=0x7f833c055480).stop 2021-03-09T09:37:29.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833802c7b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:29.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802c7b0 unknown :-1 s=CLOSED pgs=3708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:29.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:29.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:29.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:29.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009e50 con 0x7f8324003400 2021-03-09T09:37:29.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010050 con 0x7f833802d7b0 2021-03-09T09:37:29.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338070970 con 0x7f8324005e00 2021-03-09T09:37:29.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:37:29.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:29.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:37:29.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:29.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:29.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:29.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:29.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:29.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:29.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:37:29.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.974+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3714 cs=0 l=1 rev1=1 rx=0x7f833c060000 tx=0x7f833c055480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:29.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.974+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018a70 con 0x7f833802d7b0 2021-03-09T09:37:29.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.974+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01a270 con 0x7f833802d7b0 2021-03-09T09:37:29.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:29.978+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01a440 con 0x7f833802d7b0 2021-03-09T09:37:30.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:37:29 smithi146 bash[13136]: debug 2021-03-09T09:37:29.971+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10444 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:37:34.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:37:34.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:34.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3714 cs=0 l=1 rev1=1 rx=0x7f833c060000 tx=0x7f833c055480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:34.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3714 cs=0 l=1 rev1=1 rx=0x7f833c060000 tx=0x7f833c055480).stop 2021-03-09T09:37:34.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:34.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.970+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:34.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.970+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:34.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.970+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380101b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:34.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:37:34.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:34.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.970+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:34.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003480 con 0x7f8324003400 2021-03-09T09:37:34.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010050 con 0x7f8324008870 2021-03-09T09:37:34.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338029020 con 0x7f8324005e00 2021-03-09T09:37:34.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.974+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380101b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:34.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.974+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:37:34.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.974+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:34.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.974+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:34.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.974+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:34.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.974+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:34.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.974+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:34.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.974+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068790 con 0x7f8324008870 2021-03-09T09:37:34.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.974+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380101b0 secure :-1 s=READY pgs=3725 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344078300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:34.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.978+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344078190 con 0x7f8324008870 2021-03-09T09:37:34.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.978+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402beb0 con 0x7f8324008870 2021-03-09T09:37:34.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:34.982+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402abc0 con 0x7f8324008870 2021-03-09T09:37:35.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:37:34 smithi146 bash[13136]: debug 2021-03-09T09:37:34.971+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10471 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:37:39.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.973+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380101b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:37:39.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.973+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380101b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:39.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.973+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380101b0 secure :-1 s=READY pgs=3725 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344078300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:39.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.973+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380101b0 secure :-1 s=READY pgs=3725 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344078300).stop 2021-03-09T09:37:39.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380101b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:39.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380101b0 unknown :-1 s=CLOSED pgs=3725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:39.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:39.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:39.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:39.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010050 con 0x7f8324003400 2021-03-09T09:37:39.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014b00 con 0x7f833802d7b0 2021-03-09T09:37:39.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003060 con 0x7f8324005e00 2021-03-09T09:37:39.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.973+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:37:39.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.973+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:37:39.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.973+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:39.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.973+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:39.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:39.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.981+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:39.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:39.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.981+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:39.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:39.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.981+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007230 con 0x7f833802d7b0 2021-03-09T09:37:39.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3736 cs=0 l=1 rev1=1 rx=0x7f833c042da0 tx=0x7f833c055480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:39.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c042be0 con 0x7f833802d7b0 2021-03-09T09:37:39.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d290 con 0x7f833802d7b0 2021-03-09T09:37:39.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:39.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0572b0 con 0x7f833802d7b0 2021-03-09T09:37:40.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:37:39 smithi146 bash[13136]: debug 2021-03-09T09:37:39.975+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10495 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:37:44.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.973+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:37:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.973+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.973+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3736 cs=0 l=1 rev1=1 rx=0x7f833c042da0 tx=0x7f833c055480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.973+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3736 cs=0 l=1 rev1=1 rx=0x7f833c042da0 tx=0x7f833c055480).stop 2021-03-09T09:37:44.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:44.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:44.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833806cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010050 con 0x7f8324003400 2021-03-09T09:37:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e360 con 0x7f8324008870 2021-03-09T09:37:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023800 con 0x7f8324005e00 2021-03-09T09:37:44.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.973+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:37:44.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.977+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:44.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.977+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:37:44.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.977+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:44.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.977+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833806cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:44.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.977+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:44.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.977+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:44.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.977+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:44.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.977+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:44.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.977+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062880 con 0x7f8324008870 2021-03-09T09:37:44.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833806cc70 secure :-1 s=READY pgs=3744 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c055480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:44.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.981+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0294a0 con 0x7f8324008870 2021-03-09T09:37:44.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.981+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f8324008870 2021-03-09T09:37:44.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:44.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c010520 con 0x7f8324008870 2021-03-09T09:37:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:37:44 smithi146 bash[13136]: debug 2021-03-09T09:37:44.975+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10521 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:37:49.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833806cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:37:49.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833806cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:49.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833806cc70 secure :-1 s=READY pgs=3744 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c055480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:49.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833806cc70 secure :-1 s=READY pgs=3744 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c055480).stop 2021-03-09T09:37:49.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833806cc70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:49.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833806cc70 unknown :-1 s=CLOSED pgs=3744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:49.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:49.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:49.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:49.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010050 con 0x7f8324003400 2021-03-09T09:37:49.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028110 con 0x7f833802d7b0 2021-03-09T09:37:49.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013c30 con 0x7f8324005e00 2021-03-09T09:37:49.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:37:49.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:49.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:37:49.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:49.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.977+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:49.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.981+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:49.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.981+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:49.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.981+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:49.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.981+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:49.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.981+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:37:49.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.981+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3753 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344078300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:49.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.981+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344032d60 con 0x7f833802d7b0 2021-03-09T09:37:49.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833802d7b0 2021-03-09T09:37:49.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:49.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833802d7b0 2021-03-09T09:37:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:37:49 smithi146 bash[13136]: debug 2021-03-09T09:37:49.979+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10546 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:37:54.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.977+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:37:54.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.977+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:54.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.977+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3753 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344078300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:54.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.977+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3753 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344078300).stop 2021-03-09T09:37:54.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:54.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.977+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:54.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.977+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338010050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:54.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.977+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:54.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.977+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:54.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324003400 2021-03-09T09:37:54.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028110 con 0x7f8324008870 2021-03-09T09:37:54.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065e40 con 0x7f8324005e00 2021-03-09T09:37:54.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.977+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:37:54.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.977+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:54.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.981+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:37:54.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:54.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338010050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:54.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.981+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:54.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:54.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.981+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:54.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:54.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.981+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f8324008870 2021-03-09T09:37:54.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338010050 secure :-1 s=READY pgs=3759 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c055480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:54.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c058690 con 0x7f8324008870 2021-03-09T09:37:54.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8324008870 2021-03-09T09:37:54.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:54.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0572b0 con 0x7f8324008870 2021-03-09T09:37:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:37:54 smithi146 bash[13136]: debug 2021-03-09T09:37:54.979+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10572 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:37:59.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338010050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:37:59.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338010050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:59.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338010050 secure :-1 s=READY pgs=3759 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c055480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:59.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338010050 secure :-1 s=READY pgs=3759 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c055480).stop 2021-03-09T09:37:59.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338010050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:59.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338010050 unknown :-1 s=CLOSED pgs=3759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:59.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:59.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:59.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:59.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b2d0 con 0x7f8324003400 2021-03-09T09:37:59.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:37:59.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:59.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:37:59.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ff10 con 0x7f833802d7b0 2021-03-09T09:37:59.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:59.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a870 con 0x7f8324005e00 2021-03-09T09:37:59.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:59.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.985+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:59.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:59.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.985+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:59.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:59.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.985+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338013f60 con 0x7f833802d7b0 2021-03-09T09:37:59.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3767 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c055480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:59.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c040ac0 con 0x7f833802d7b0 2021-03-09T09:37:59.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0792c0 con 0x7f833802d7b0 2021-03-09T09:37:59.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:37:59.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0104f0 con 0x7f833802d7b0 2021-03-09T09:38:00.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:37:59 smithi146 bash[13136]: debug 2021-03-09T09:37:59.983+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10596 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:38:04.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:38:04.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:04.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3767 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c055480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:04.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3767 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c055480).stop 2021-03-09T09:38:04.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:04.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338010990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:04.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023800 con 0x7f8324003400 2021-03-09T09:38:04.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324008870 2021-03-09T09:38:04.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020460 con 0x7f8324005e00 2021-03-09T09:38:04.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:38:04.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:38:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338010990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:04.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:04.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:04.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:04.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:04.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f8324008870 2021-03-09T09:38:04.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338010990 secure :-1 s=READY pgs=3775 cs=0 l=1 rev1=1 rx=0x7f833c042040 tx=0x7f833c05ff20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c023000 con 0x7f8324008870 2021-03-09T09:38:04.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c023000 con 0x7f8324008870 2021-03-09T09:38:04.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:04.993+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06a2d0 con 0x7f8324008870 2021-03-09T09:38:05.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:38:04 smithi146 bash[13136]: debug 2021-03-09T09:38:04.983+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10623 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:38:09.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338010990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:38:09.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338010990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:09.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338010990 secure :-1 s=READY pgs=3775 cs=0 l=1 rev1=1 rx=0x7f833c042040 tx=0x7f833c05ff20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:09.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338010990 secure :-1 s=READY pgs=3775 cs=0 l=1 rev1=1 rx=0x7f833c042040 tx=0x7f833c05ff20).stop 2021-03-09T09:38:09.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338010990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:09.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338010990 unknown :-1 s=CLOSED pgs=3775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:09.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:09.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:09.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:09.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380149a0 con 0x7f8324003400 2021-03-09T09:38:09.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019f10 con 0x7f833802d7b0 2021-03-09T09:38:09.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011510 con 0x7f8324005e00 2021-03-09T09:38:09.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:38:09.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:38:09.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:09.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:09.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:09.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:09.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:09.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:09.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:09.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007280 con 0x7f833802d7b0 2021-03-09T09:38:09.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3787 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c05ff20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:09.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.993+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c06a4e0 con 0x7f833802d7b0 2021-03-09T09:38:09.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c023000 con 0x7f833802d7b0 2021-03-09T09:38:10.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:09.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06a3b0 con 0x7f833802d7b0 2021-03-09T09:38:10.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:38:09 smithi146 bash[13136]: debug 2021-03-09T09:38:09.987+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10647 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:38:14.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.985+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:38:14.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.985+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:14.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3787 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c05ff20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:14.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3787 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c05ff20).stop 2021-03-09T09:38:14.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:14.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:14.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338013f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:14.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:14.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:38:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004a20 con 0x7f8324008870 2021-03-09T09:38:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a870 con 0x7f8324005e00 2021-03-09T09:38:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:38:14.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:14.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:38:14.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:14.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338013f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:14.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:14.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:14.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:14.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:14.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338009a10 con 0x7f8324008870 2021-03-09T09:38:14.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.993+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338013f20 secure :-1 s=READY pgs=3792 cs=0 l=1 rev1=1 rx=0x7f833c043050 tx=0x7f833c05ff20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:14.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.993+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c053d20 con 0x7f8324008870 2021-03-09T09:38:14.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.993+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c078220 con 0x7f8324008870 2021-03-09T09:38:14.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:14.993+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054310 con 0x7f8324008870 2021-03-09T09:38:15.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:38:14 smithi146 bash[13136]: debug 2021-03-09T09:38:14.991+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10674 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:38:19.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338013f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:38:19.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338013f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:19.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338013f20 secure :-1 s=READY pgs=3792 cs=0 l=1 rev1=1 rx=0x7f833c043050 tx=0x7f833c05ff20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:19.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338013f20 secure :-1 s=READY pgs=3792 cs=0 l=1 rev1=1 rx=0x7f833c043050 tx=0x7f833c05ff20).stop 2021-03-09T09:38:19.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338013f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:19.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338013f20 unknown :-1 s=CLOSED pgs=3792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:19.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:19.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:19.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:19.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019f10 con 0x7f8324003400 2021-03-09T09:38:19.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f833802d7b0 2021-03-09T09:38:19.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023800 con 0x7f8324005e00 2021-03-09T09:38:19.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:38:19.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:19.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:38:19.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:19.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:19.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.993+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:19.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.993+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:19.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.993+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:19.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.993+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:20.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.993+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801c7d0 con 0x7f833802d7b0 2021-03-09T09:38:20.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.993+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3802 cs=0 l=1 rev1=1 rx=0x7f833c053cb0 tx=0x7f833c05ff20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:20.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.993+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01cb80 con 0x7f833802d7b0 2021-03-09T09:38:20.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.993+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01cd50 con 0x7f833802d7b0 2021-03-09T09:38:20.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:19.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054380 con 0x7f833802d7b0 2021-03-09T09:38:20.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:38:19 smithi146 bash[13136]: debug 2021-03-09T09:38:19.991+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10698 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:38:24.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:38:24.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:24.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3802 cs=0 l=1 rev1=1 rx=0x7f833c053cb0 tx=0x7f833c05ff20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:24.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3802 cs=0 l=1 rev1=1 rx=0x7f833c053cb0 tx=0x7f833c05ff20).stop 2021-03-09T09:38:24.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:24.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:24.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:24.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338013f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:24.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:24.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338013f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:24.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:38:24.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:24.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:38:24.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:24.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:38:25.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380162f0 con 0x7f8324008870 2021-03-09T09:38:25.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065ce0 con 0x7f8324005e00 2021-03-09T09:38:25.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.993+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:25.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.997+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:25.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.997+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:25.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.997+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:25.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.997+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338069910 con 0x7f8324008870 2021-03-09T09:38:25.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.997+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338013f20 secure :-1 s=READY pgs=3810 cs=0 l=1 rev1=1 rx=0x7f8344013410 tx=0x7f8344078300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:25.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f8324008870 2021-03-09T09:38:25.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:24.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T09:38:25.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:25.001+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8324008870 2021-03-09T09:38:25.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:38:25 smithi146 bash[13136]: debug 2021-03-09T09:38:24.995+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10725 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:38:29.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338013f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:38:30.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338013f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:30.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338013f20 secure :-1 s=READY pgs=3810 cs=0 l=1 rev1=1 rx=0x7f8344013410 tx=0x7f8344078300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:30.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338013f20 secure :-1 s=READY pgs=3810 cs=0 l=1 rev1=1 rx=0x7f8344013410 tx=0x7f8344078300).stop 2021-03-09T09:38:30.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338013f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:30.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338013f20 unknown :-1 s=CLOSED pgs=3810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:30.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:30.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:30.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:30.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e230 con 0x7f8324003400 2021-03-09T09:38:30.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e0f0 con 0x7f833802d7b0 2021-03-09T09:38:30.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e920 con 0x7f8324005e00 2021-03-09T09:38:30.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:38:30.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:30.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:38:30.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:30.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.992+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:30.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.996+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:30.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.996+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:30.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.996+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:30.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.996+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:30.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.996+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:38:30.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.996+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3818 cs=0 l=1 rev1=1 rx=0x7f833c0555e0 tx=0x7f833c05ff20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:30.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f833802d7b0 2021-03-09T09:38:30.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:29.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:38:30.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:30.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0572b0 con 0x7f833802d7b0 2021-03-09T09:38:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:38:29 smithi146 bash[13136]: debug 2021-03-09T09:38:29.995+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10749 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:38:35.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:38:35.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:35.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3818 cs=0 l=1 rev1=1 rx=0x7f833c0555e0 tx=0x7f833c05ff20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:35.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3818 cs=0 l=1 rev1=1 rx=0x7f833c0555e0 tx=0x7f833c05ff20).stop 2021-03-09T09:38:35.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:35.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:35.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:35.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:35.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:35.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016d70 con 0x7f8324003400 2021-03-09T09:38:35.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b40 con 0x7f8324008870 2021-03-09T09:38:35.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c810 con 0x7f8324005e00 2021-03-09T09:38:35.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:38:35.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:35.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:38:35.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:35.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:34.996+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:35.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:35.000+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:35.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:35.000+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:35.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:35.000+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:35.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:35.000+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:35.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:35.000+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026710 con 0x7f8324008870 2021-03-09T09:38:35.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:35.000+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a7d0 secure :-1 s=READY pgs=3829 cs=0 l=1 rev1=1 rx=0x7f833c06a8a0 tx=0x7f833c05ff20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:35.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:35.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c06a6e0 con 0x7f8324008870 2021-03-09T09:38:35.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:35.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f030 con 0x7f8324008870 2021-03-09T09:38:35.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:35.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c034d70 con 0x7f8324008870 2021-03-09T09:38:35.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:38:35 smithi146 bash[13136]: debug 2021-03-09T09:38:34.998+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10776 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:38:40.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:39.996+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:38:40.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:39.996+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:40.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:39.996+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a7d0 secure :-1 s=READY pgs=3829 cs=0 l=1 rev1=1 rx=0x7f833c06a8a0 tx=0x7f833c05ff20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:40.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:39.996+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a7d0 secure :-1 s=READY pgs=3829 cs=0 l=1 rev1=1 rx=0x7f833c06a8a0 tx=0x7f833c05ff20).stop 2021-03-09T09:38:40.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:39.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800a7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:40.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:39.996+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a7d0 unknown :-1 s=CLOSED pgs=3829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:40.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:39.996+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:40.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:39.996+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:40.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:39.996+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:40.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:39.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b40 con 0x7f8324003400 2021-03-09T09:38:40.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:39.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a900 con 0x7f833802d7b0 2021-03-09T09:38:40.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:39.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e230 con 0x7f8324005e00 2021-03-09T09:38:40.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:39.996+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:38:40.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:39.996+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:40.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:39.996+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:38:40.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:39.996+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:40.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:40.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:40.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:40.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:40.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:40.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:40.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:40.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:40.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:40.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:40.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:40.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:38:40.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:40.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3841 cs=0 l=1 rev1=1 rx=0x7f833c079d00 tx=0x7f833c05ff20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:40.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:40.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c06a2d0 con 0x7f833802d7b0 2021-03-09T09:38:40.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:40.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:38:40.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:40.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833802d7b0 2021-03-09T09:38:40.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:38:40 smithi146 bash[13136]: debug 2021-03-09T09:38:39.998+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10800 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:38:45.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:38:45.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:45.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3841 cs=0 l=1 rev1=1 rx=0x7f833c079d00 tx=0x7f833c05ff20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:45.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3841 cs=0 l=1 rev1=1 rx=0x7f833c079d00 tx=0x7f833c05ff20).stop 2021-03-09T09:38:45.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:45.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:45.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:45.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:45.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:45.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019320 con 0x7f8324003400 2021-03-09T09:38:45.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026760 con 0x7f8324008870 2021-03-09T09:38:45.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380124e0 con 0x7f8324005e00 2021-03-09T09:38:45.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:38:45.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:45.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:38:45.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:45.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.000+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:45.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.004+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:45.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.004+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:45.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.004+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:45.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.004+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:45.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.004+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380064d0 con 0x7f8324008870 2021-03-09T09:38:45.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.004+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a7d0 secure :-1 s=READY pgs=3844 cs=0 l=1 rev1=1 rx=0x7f833c0554c0 tx=0x7f833c05ff20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:45.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c078aa0 con 0x7f8324008870 2021-03-09T09:38:45.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8324008870 2021-03-09T09:38:45.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:45.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c071240 con 0x7f8324008870 2021-03-09T09:38:45.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:38:45 smithi146 bash[13136]: debug 2021-03-09T09:38:45.002+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10827 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:38:50.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.000+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:38:50.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.000+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:50.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.000+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a7d0 secure :-1 s=READY pgs=3844 cs=0 l=1 rev1=1 rx=0x7f833c0554c0 tx=0x7f833c05ff20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:50.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.000+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a7d0 secure :-1 s=READY pgs=3844 cs=0 l=1 rev1=1 rx=0x7f833c0554c0 tx=0x7f833c05ff20).stop 2021-03-09T09:38:50.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800a7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:50.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a7d0 unknown :-1 s=CLOSED pgs=3844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:50.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:50.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:50.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:50.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a1f0 con 0x7f8324003400 2021-03-09T09:38:50.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019320 con 0x7f833802d7b0 2021-03-09T09:38:50.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380124e0 con 0x7f8324005e00 2021-03-09T09:38:50.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:38:50.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:50.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:38:50.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:50.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:50.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:50.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:50.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:50.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:50.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.004+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024190 con 0x7f833802d7b0 2021-03-09T09:38:50.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3856 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c05ff20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:50.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00f9c0 con 0x7f833802d7b0 2021-03-09T09:38:50.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c078aa0 con 0x7f833802d7b0 2021-03-09T09:38:50.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:50.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05cce0 con 0x7f833802d7b0 2021-03-09T09:38:50.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:38:50 smithi146 bash[13136]: debug 2021-03-09T09:38:50.002+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10851 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:38:55.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:38:55.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:55.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3856 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c05ff20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:55.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3856 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c05ff20).stop 2021-03-09T09:38:55.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:55.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:55.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338002280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:55.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:55.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:55.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380113b0 con 0x7f8324003400 2021-03-09T09:38:55.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069130 con 0x7f8324008870 2021-03-09T09:38:55.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069890 con 0x7f8324005e00 2021-03-09T09:38:55.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:38:55.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:38:55.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:55.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:55.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338002280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:55.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:55.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:55.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:55.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:55.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f8324008870 2021-03-09T09:38:55.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338002280 secure :-1 s=READY pgs=3862 cs=0 l=1 rev1=1 rx=0x7f833c035520 tx=0x7f833c05ff20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:55.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01a7b0 con 0x7f8324008870 2021-03-09T09:38:55.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f8324008870 2021-03-09T09:38:55.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:38:55.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c042df0 con 0x7f8324008870 2021-03-09T09:38:55.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:38:55 smithi146 bash[13136]: debug 2021-03-09T09:38:55.006+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10878 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:39:00.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.004+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338002280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:39:00.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.004+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338002280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:00.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.004+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338002280 secure :-1 s=READY pgs=3862 cs=0 l=1 rev1=1 rx=0x7f833c035520 tx=0x7f833c05ff20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:00.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.004+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338002280 secure :-1 s=READY pgs=3862 cs=0 l=1 rev1=1 rx=0x7f833c035520 tx=0x7f833c05ff20).stop 2021-03-09T09:39:00.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338002280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:00.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.004+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338002280 unknown :-1 s=CLOSED pgs=3862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:00.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.004+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:00.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:00.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.008+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:39:00.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:39:00.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.008+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:00.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:00.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:00.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069890 con 0x7f8324003400 2021-03-09T09:39:00.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065ca0 con 0x7f833802d7b0 2021-03-09T09:39:00.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013d80 con 0x7f8324005e00 2021-03-09T09:39:00.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:00.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:00.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:00.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:00.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:00.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:39:00.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.012+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3872 cs=0 l=1 rev1=1 rx=0x7f833c047740 tx=0x7f833c05ff20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:00.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0185f0 con 0x7f833802d7b0 2021-03-09T09:39:00.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:39:00.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:00.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833802d7b0 2021-03-09T09:39:00.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:39:00 smithi146 bash[13136]: debug 2021-03-09T09:39:00.010+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10902 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:39:05.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:39:05.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:05.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3872 cs=0 l=1 rev1=1 rx=0x7f833c047740 tx=0x7f833c05ff20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:05.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3872 cs=0 l=1 rev1=1 rx=0x7f833c047740 tx=0x7f833c05ff20).stop 2021-03-09T09:39:05.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338002280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a900 con 0x7f8324003400 2021-03-09T09:39:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072b30 con 0x7f8324008870 2021-03-09T09:39:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069890 con 0x7f8324005e00 2021-03-09T09:39:05.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:39:05.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:05.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:39:05.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:05.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338002280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:05.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:05.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:05.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:05.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:05.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338021e30 con 0x7f8324008870 2021-03-09T09:39:05.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.012+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338002280 secure :-1 s=READY pgs=3883 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344078300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:05.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f8324008870 2021-03-09T09:39:05.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T09:39:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:05.020+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344049710 con 0x7f8324008870 2021-03-09T09:39:05.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:39:05 smithi146 bash[13136]: debug 2021-03-09T09:39:05.010+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10929 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:39:10.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.008+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338002280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:39:10.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.008+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338002280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:10.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.008+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338002280 secure :-1 s=READY pgs=3883 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344078300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:10.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.008+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338002280 secure :-1 s=READY pgs=3883 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344078300).stop 2021-03-09T09:39:10.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338002280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:10.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338002280 unknown :-1 s=CLOSED pgs=3883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:10.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:10.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:10.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:10.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c350 con 0x7f8324003400 2021-03-09T09:39:10.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072b30 con 0x7f833802d7b0 2021-03-09T09:39:10.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4a0 con 0x7f8324005e00 2021-03-09T09:39:10.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:39:10.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:10.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:39:10.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:10.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:10.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:10.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:10.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:10.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:10.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024190 con 0x7f833802d7b0 2021-03-09T09:39:10.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3888 cs=0 l=1 rev1=1 rx=0x7f833c071450 tx=0x7f833c05ff20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:10.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c069960 con 0x7f833802d7b0 2021-03-09T09:39:10.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01cec0 con 0x7f833802d7b0 2021-03-09T09:39:10.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:10.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0572b0 con 0x7f833802d7b0 2021-03-09T09:39:10.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:39:10 smithi146 bash[13136]: debug 2021-03-09T09:39:10.010+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10953 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:39:15.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:39:15.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:15.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3888 cs=0 l=1 rev1=1 rx=0x7f833c071450 tx=0x7f833c05ff20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:15.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3888 cs=0 l=1 rev1=1 rx=0x7f833c071450 tx=0x7f833c05ff20).stop 2021-03-09T09:39:15.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:15.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:15.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:15.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:15.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:15.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a1e0 con 0x7f8324003400 2021-03-09T09:39:15.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d660 con 0x7f8324008870 2021-03-09T09:39:15.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fb70 con 0x7f8324005e00 2021-03-09T09:39:15.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:39:15.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:39:15.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:15.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:15.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:15.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:15.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:15.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:15.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:15.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f8324008870 2021-03-09T09:39:15.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 secure :-1 s=READY pgs=3896 cs=0 l=1 rev1=1 rx=0x7f833c070030 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:15.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0258e0 con 0x7f8324008870 2021-03-09T09:39:15.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c069960 con 0x7f8324008870 2021-03-09T09:39:15.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:15.020+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c029c50 con 0x7f8324008870 2021-03-09T09:39:15.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:39:15 smithi146 bash[13136]: debug 2021-03-09T09:39:15.014+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 10980 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:39:20.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338023140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:39:20.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338023140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:20.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 secure :-1 s=READY pgs=3896 cs=0 l=1 rev1=1 rx=0x7f833c070030 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:20.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 secure :-1 s=READY pgs=3896 cs=0 l=1 rev1=1 rx=0x7f833c070030 tx=0x7f833c04de90).stop 2021-03-09T09:39:20.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338023140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:20.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 unknown :-1 s=CLOSED pgs=3896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:20.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:20.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:20.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:39:20.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:20.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:20.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:39:20.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064ed0 con 0x7f833802d7b0 2021-03-09T09:39:20.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380124e0 con 0x7f8324005e00 2021-03-09T09:39:20.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:39:20.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:20.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:20.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:20.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:20.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:20.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:20.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.015+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:39:20.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.019+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3907 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f8344078300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:20.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.019+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f833802d7b0 2021-03-09T09:39:20.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.019+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833802d7b0 2021-03-09T09:39:20.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:20.019+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833802d7b0 2021-03-09T09:39:20.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:39:20 smithi146 bash[13136]: debug 2021-03-09T09:39:20.018+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11004 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:39:25.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:39:25.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:25.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3907 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f8344078300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:25.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3907 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f8344078300).stop 2021-03-09T09:39:25.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:25.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:25.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:25.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:25.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:25.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064ed0 con 0x7f8324003400 2021-03-09T09:39:25.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d8c0 con 0x7f8324008870 2021-03-09T09:39:25.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806b180 con 0x7f8324005e00 2021-03-09T09:39:25.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:39:25.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:25.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:25.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:39:25.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.019+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:25.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.023+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.023+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.023+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.023+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.023+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f8324008870 2021-03-09T09:39:25.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.023+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=3914 cs=0 l=1 rev1=1 rx=0x7f833c01a5b0 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:25.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.023+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0218d0 con 0x7f8324008870 2021-03-09T09:39:25.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.023+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c055310 con 0x7f8324008870 2021-03-09T09:39:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:25.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8324008870 2021-03-09T09:39:25.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:39:25 smithi146 bash[13136]: debug 2021-03-09T09:39:25.018+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11030 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:39:30.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:39:30.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:30.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=3914 cs=0 l=1 rev1=1 rx=0x7f833c01a5b0 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:30.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=3914 cs=0 l=1 rev1=1 rx=0x7f833c01a5b0 tx=0x7f833c04de90).stop 2021-03-09T09:39:30.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:30.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=CLOSED pgs=3914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:30.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:30.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:30.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:30.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065ce0 con 0x7f8324003400 2021-03-09T09:39:30.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d8c0 con 0x7f833802d7b0 2021-03-09T09:39:30.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f0a0 con 0x7f8324005e00 2021-03-09T09:39:30.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:39:30.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:30.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:39:30.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:30.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:30.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:30.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:30.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:30.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:30.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:39:30.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.023+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3922 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344078300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:30.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405ce80 con 0x7f833802d7b0 2021-03-09T09:39:30.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400fbc0 con 0x7f833802d7b0 2021-03-09T09:39:30.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:30.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833802d7b0 2021-03-09T09:39:30.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:39:30 smithi146 bash[13136]: debug 2021-03-09T09:39:30.022+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11054 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:39:35.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.023+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:39:35.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.023+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:35.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.023+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3922 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344078300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:35.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.023+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3922 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344078300).stop 2021-03-09T09:39:35.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.023+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:35.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.023+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:35.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.023+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:35.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.023+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:35.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.023+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:35.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.023+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067670 con 0x7f8324003400 2021-03-09T09:39:35.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.023+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019f10 con 0x7f8324008870 2021-03-09T09:39:35.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.023+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069890 con 0x7f8324005e00 2021-03-09T09:39:35.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.023+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:39:35.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.023+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:35.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.023+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:39:35.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.023+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:35.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.027+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:35.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.027+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:35.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.027+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:35.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.027+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:35.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.027+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:35.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.027+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f8324008870 2021-03-09T09:39:35.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.027+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=3934 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:35.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c060c30 con 0x7f8324008870 2021-03-09T09:39:35.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c060e60 con 0x7f8324008870 2021-03-09T09:39:35.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:35.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055310 con 0x7f8324008870 2021-03-09T09:39:35.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:39:35 smithi146 bash[13136]: debug 2021-03-09T09:39:35.022+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11081 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:39:40.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.027+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:39:40.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.027+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:40.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.027+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=3934 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:40.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.027+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=3934 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04de90).stop 2021-03-09T09:39:40.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:40.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.027+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=CLOSED pgs=3934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:40.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.027+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:40.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.027+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:40.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.027+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:40.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067670 con 0x7f8324003400 2021-03-09T09:39:40.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019f10 con 0x7f833802d7b0 2021-03-09T09:39:40.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e320 con 0x7f8324005e00 2021-03-09T09:39:40.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.027+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:39:40.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.027+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:40.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.027+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:39:40.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.027+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:40.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.031+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:40.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.031+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:40.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.031+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:40.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.031+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:40.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.031+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:40.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.031+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:39:40.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.031+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3941 cs=0 l=1 rev1=1 rx=0x7f833c0797a0 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:40.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c054310 con 0x7f833802d7b0 2021-03-09T09:39:40.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c054470 con 0x7f833802d7b0 2021-03-09T09:39:40.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:40.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c062350 con 0x7f833802d7b0 2021-03-09T09:39:40.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:39:40 smithi146 bash[13136]: debug 2021-03-09T09:39:40.026+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11115 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:39:45.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.027+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:39:45.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.027+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:45.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.027+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3941 cs=0 l=1 rev1=1 rx=0x7f833c0797a0 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:45.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.027+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3941 cs=0 l=1 rev1=1 rx=0x7f833c0797a0 tx=0x7f833c04de90).stop 2021-03-09T09:39:45.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:45.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.027+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:45.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.027+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:45.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.027+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:45.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.027+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:45.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e320 con 0x7f8324003400 2021-03-09T09:39:45.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380667f0 con 0x7f8324008870 2021-03-09T09:39:45.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380698a0 con 0x7f8324005e00 2021-03-09T09:39:45.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.027+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:39:45.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.027+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:45.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.027+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:39:45.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.027+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:45.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.031+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:45.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.031+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:45.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.031+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:45.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.031+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:45.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.031+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:45.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.031+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f8324008870 2021-03-09T09:39:45.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.031+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=3950 cs=0 l=1 rev1=1 rx=0x7f8344003fd0 tx=0x7f8344078300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:45.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f8324008870 2021-03-09T09:39:45.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T09:39:45.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:45.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8324008870 2021-03-09T09:39:45.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:39:45 smithi146 bash[13136]: debug 2021-03-09T09:39:45.026+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11131 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:39:50.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:39:50.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:50.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=3950 cs=0 l=1 rev1=1 rx=0x7f8344003fd0 tx=0x7f8344078300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:50.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=3950 cs=0 l=1 rev1=1 rx=0x7f8344003fd0 tx=0x7f8344078300).stop 2021-03-09T09:39:50.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:50.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=CLOSED pgs=3950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:50.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:50.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:50.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:50.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067670 con 0x7f8324003400 2021-03-09T09:39:50.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e320 con 0x7f833802d7b0 2021-03-09T09:39:50.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380104f0 con 0x7f8324005e00 2021-03-09T09:39:50.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:39:50.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:50.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:39:50.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:50.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.031+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:50.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.035+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:50.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.035+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:50.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.035+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:50.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.035+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:50.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.035+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012a40 con 0x7f833802d7b0 2021-03-09T09:39:50.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.035+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3957 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344078300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:50.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400f3c0 con 0x7f833802d7b0 2021-03-09T09:39:50.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f833802d7b0 2021-03-09T09:39:50.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:50.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834406b3f0 con 0x7f833802d7b0 2021-03-09T09:39:50.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:39:50 smithi146 bash[13136]: debug 2021-03-09T09:39:50.029+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11158 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:39:55.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.031+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:39:55.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.031+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:55.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.031+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3957 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344078300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:55.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.031+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3957 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344078300).stop 2021-03-09T09:39:55.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:55.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.031+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:55.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.031+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:55.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.031+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:55.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.031+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:55.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380104f0 con 0x7f8324003400 2021-03-09T09:39:55.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020460 con 0x7f8324008870 2021-03-09T09:39:55.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e320 con 0x7f8324005e00 2021-03-09T09:39:55.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.031+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:39:55.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.031+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:39:55.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.031+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:55.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.035+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:55.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.035+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:55.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.035+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:55.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.035+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:55.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.035+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:55.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.035+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:55.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.035+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f8324008870 2021-03-09T09:39:55.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.035+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=3968 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:55.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01cec0 con 0x7f8324008870 2021-03-09T09:39:55.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8324008870 2021-03-09T09:39:55.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:39:55.043+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055310 con 0x7f8324008870 2021-03-09T09:39:55.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:39:55 smithi146 bash[13136]: debug 2021-03-09T09:39:55.033+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11182 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:40:00.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.035+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:40:00.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.035+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:00.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.035+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=3968 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:00.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.035+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=3968 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c04de90).stop 2021-03-09T09:40:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.035+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=CLOSED pgs=3968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.035+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.035+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:00.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.035+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:00.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380667f0 con 0x7f8324003400 2021-03-09T09:40:00.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014e40 con 0x7f833802d7b0 2021-03-09T09:40:00.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d660 con 0x7f8324005e00 2021-03-09T09:40:00.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.035+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:40:00.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.035+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:00.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.035+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:40:00.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.035+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:00.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:00.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:00.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:00.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:00.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:00.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380628c0 con 0x7f833802d7b0 2021-03-09T09:40:00.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3975 cs=0 l=1 rev1=1 rx=0x7f833c009980 tx=0x7f833c000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:00.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.043+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c057d60 con 0x7f833802d7b0 2021-03-09T09:40:00.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.043+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833802d7b0 2021-03-09T09:40:00.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:00.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c029c50 con 0x7f833802d7b0 2021-03-09T09:40:00.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:40:00 smithi146 bash[13136]: debug 2021-03-09T09:40:00.033+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11213 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:40:05.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:40:05.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:05.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3975 cs=0 l=1 rev1=1 rx=0x7f833c009980 tx=0x7f833c000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:05.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3975 cs=0 l=1 rev1=1 rx=0x7f833c009980 tx=0x7f833c000bc0).stop 2021-03-09T09:40:05.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:05.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:05.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:05.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:05.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380698a0 con 0x7f8324003400 2021-03-09T09:40:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010dd0 con 0x7f8324008870 2021-03-09T09:40:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806f020 con 0x7f8324005e00 2021-03-09T09:40:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:40:05.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:40:05.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:05.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:05.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:05.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:05.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:05.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:05.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:05.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012a40 con 0x7f8324008870 2021-03-09T09:40:05.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.043+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=3983 cs=0 l=1 rev1=1 rx=0x7f833c062a30 tx=0x7f833c000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:05.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.043+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c06aac0 con 0x7f8324008870 2021-03-09T09:40:05.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.043+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c003680 con 0x7f8324008870 2021-03-09T09:40:05.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:05.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8324008870 2021-03-09T09:40:05.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:40:05 smithi146 bash[13136]: debug 2021-03-09T09:40:05.037+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11233 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:40:10.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.038+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:40:10.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.038+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:10.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.038+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=3983 cs=0 l=1 rev1=1 rx=0x7f833c062a30 tx=0x7f833c000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:10.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.038+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=3983 cs=0 l=1 rev1=1 rx=0x7f833c062a30 tx=0x7f833c000bc0).stop 2021-03-09T09:40:10.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.038+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:10.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.038+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=CLOSED pgs=3983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:10.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.038+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:10.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.038+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:10.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.038+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:10.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.038+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014e40 con 0x7f8324003400 2021-03-09T09:40:10.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.038+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4a0 con 0x7f833802d7b0 2021-03-09T09:40:10.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.038+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c350 con 0x7f8324005e00 2021-03-09T09:40:10.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.038+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:40:10.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.038+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:10.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.038+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:40:10.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.038+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:10.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.042+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:10.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.046+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:10.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.046+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.046+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.046+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.046+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f833802d7b0 2021-03-09T09:40:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.046+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3995 cs=0 l=1 rev1=1 rx=0x7f834405d050 tx=0x7f8344078300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:10.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.046+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405ce80 con 0x7f833802d7b0 2021-03-09T09:40:10.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.046+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f833802d7b0 2021-03-09T09:40:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:10.050+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833802d7b0 2021-03-09T09:40:10.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:40:10 smithi146 bash[13136]: debug 2021-03-09T09:40:10.037+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11259 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:40:15.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:40:15.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:15.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3995 cs=0 l=1 rev1=1 rx=0x7f834405d050 tx=0x7f8344078300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:15.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=3995 cs=0 l=1 rev1=1 rx=0x7f834405d050 tx=0x7f8344078300).stop 2021-03-09T09:40:15.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:15.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=3995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:15.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:15.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:15.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:15.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016f00 con 0x7f8324003400 2021-03-09T09:40:15.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806f020 con 0x7f8324008870 2021-03-09T09:40:15.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d660 con 0x7f8324005e00 2021-03-09T09:40:15.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:15.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:40:15.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:15.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:40:15.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:15.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:15.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.042+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:15.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.046+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:15.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.046+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:15.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.046+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f8324008870 2021-03-09T09:40:15.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.046+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=4001 cs=0 l=1 rev1=1 rx=0x7f833800df30 tx=0x7f8344078300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:15.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.046+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344029070 con 0x7f8324008870 2021-03-09T09:40:15.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.046+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033040 con 0x7f8324008870 2021-03-09T09:40:15.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:15.050+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f8324008870 2021-03-09T09:40:15.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:40:15 smithi146 bash[13136]: debug 2021-03-09T09:40:15.041+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11284 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:40:20.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:40:20.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:20.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=4001 cs=0 l=1 rev1=1 rx=0x7f833800df30 tx=0x7f8344078300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:20.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=4001 cs=0 l=1 rev1=1 rx=0x7f833800df30 tx=0x7f8344078300).stop 2021-03-09T09:40:20.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:20.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=CLOSED pgs=4001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:20.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:20.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:20.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:20.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060980 con 0x7f8324003400 2021-03-09T09:40:20.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380157e0 con 0x7f833802d7b0 2021-03-09T09:40:20.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017e40 con 0x7f8324005e00 2021-03-09T09:40:20.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:40:20.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:20.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:20.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:40:20.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:20.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:20.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:20.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:20.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:20.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e550 con 0x7f833802d7b0 2021-03-09T09:40:20.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.046+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4009 cs=0 l=1 rev1=1 rx=0x7f833c023d00 tx=0x7f833c000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:20.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.050+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01cec0 con 0x7f833802d7b0 2021-03-09T09:40:20.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.050+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:40:20.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:20.054+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055310 con 0x7f833802d7b0 2021-03-09T09:40:20.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:40:20 smithi146 bash[13136]: debug 2021-03-09T09:40:20.045+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11318 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:40:25.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.046+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:40:25.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.046+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:25.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.046+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4009 cs=0 l=1 rev1=1 rx=0x7f833c023d00 tx=0x7f833c000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:25.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.046+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4009 cs=0 l=1 rev1=1 rx=0x7f833c023d00 tx=0x7f833c000bc0).stop 2021-03-09T09:40:25.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.046+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:25.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.046+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:25.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.046+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:25.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.046+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:25.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.046+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:25.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.046+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c230 con 0x7f8324003400 2021-03-09T09:40:25.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.046+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d660 con 0x7f8324008870 2021-03-09T09:40:25.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.046+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064ed0 con 0x7f8324005e00 2021-03-09T09:40:25.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.046+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:40:25.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.046+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:25.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.050+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:40:25.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.050+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:25.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.050+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:25.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.050+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:25.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.050+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:25.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.050+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:25.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.050+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:25.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.050+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f8324008870 2021-03-09T09:40:25.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.050+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=4019 cs=0 l=1 rev1=1 rx=0x7f833c0799c0 tx=0x7f833c000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:25.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.054+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c054310 con 0x7f8324008870 2021-03-09T09:40:25.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.058+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8324008870 2021-03-09T09:40:25.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:25.058+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0543f0 con 0x7f8324008870 2021-03-09T09:40:25.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:40:25 smithi146 bash[13136]: debug 2021-03-09T09:40:25.045+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11343 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:40:30.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:40:30.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:30.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=4019 cs=0 l=1 rev1=1 rx=0x7f833c0799c0 tx=0x7f833c000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:30.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 secure :-1 s=READY pgs=4019 cs=0 l=1 rev1=1 rx=0x7f833c0799c0 tx=0x7f833c000bc0).stop 2021-03-09T09:40:30.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:30.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005380 unknown :-1 s=CLOSED pgs=4019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:30.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:30.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:30.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:30.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c230 con 0x7f8324003400 2021-03-09T09:40:30.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dbb0 con 0x7f833802d7b0 2021-03-09T09:40:30.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802dd80 con 0x7f8324005e00 2021-03-09T09:40:30.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:40:30.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:30.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:40:30.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:30.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.050+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:30.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.054+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:30.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.054+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:30.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.054+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:30.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.054+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:30.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.054+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:40:30.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.054+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4026 cs=0 l=1 rev1=1 rx=0x7f833c018620 tx=0x7f833c060d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:30.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.054+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018460 con 0x7f833802d7b0 2021-03-09T09:40:30.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.054+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f833802d7b0 2021-03-09T09:40:30.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:30.058+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054310 con 0x7f833802d7b0 2021-03-09T09:40:30.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:40:30 smithi146 bash[13136]: debug 2021-03-09T09:40:30.049+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11361 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:40:35.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.050+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:40:35.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.050+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:35.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.050+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4026 cs=0 l=1 rev1=1 rx=0x7f833c018620 tx=0x7f833c060d60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:35.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.050+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4026 cs=0 l=1 rev1=1 rx=0x7f833c018620 tx=0x7f833c060d60).stop 2021-03-09T09:40:35.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.050+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:35.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.050+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:35.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.050+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:35.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.050+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:35.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.050+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:35.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.050+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c230 con 0x7f8324003400 2021-03-09T09:40:35.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.050+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380157e0 con 0x7f8324008870 2021-03-09T09:40:35.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.050+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d040 con 0x7f8324005e00 2021-03-09T09:40:35.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.050+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:40:35.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.050+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:35.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.050+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:40:35.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.050+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:35.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.054+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:35.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.054+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:35.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.054+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:35.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.054+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:35.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.054+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:35.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.054+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338019100 con 0x7f8324008870 2021-03-09T09:40:35.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.054+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007540 secure :-1 s=READY pgs=4037 cs=0 l=1 rev1=1 rx=0x7f833c053840 tx=0x7f833c060d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:35.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.058+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c060e20 con 0x7f8324008870 2021-03-09T09:40:35.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.058+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f8324008870 2021-03-09T09:40:35.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:35.062+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8324008870 2021-03-09T09:40:35.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:40:35 smithi146 bash[13136]: debug 2021-03-09T09:40:35.049+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11386 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:40:40.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:40:40.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:40.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007540 secure :-1 s=READY pgs=4037 cs=0 l=1 rev1=1 rx=0x7f833c053840 tx=0x7f833c060d60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:40.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007540 secure :-1 s=READY pgs=4037 cs=0 l=1 rev1=1 rx=0x7f833c053840 tx=0x7f833c060d60).stop 2021-03-09T09:40:40.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:40.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007540 unknown :-1 s=CLOSED pgs=4037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:40.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:40.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:40.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:40.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806f020 con 0x7f8324003400 2021-03-09T09:40:40.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b050 con 0x7f833802d7b0 2021-03-09T09:40:40.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380104f0 con 0x7f8324005e00 2021-03-09T09:40:40.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:40:40.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:40.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:40:40.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:40.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:40.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:40.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:40.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:40.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:40.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.054+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380628c0 con 0x7f833802d7b0 2021-03-09T09:40:40.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.058+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4044 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c060d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:40.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.058+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c06a4b0 con 0x7f833802d7b0 2021-03-09T09:40:40.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.058+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c06a2d0 con 0x7f833802d7b0 2021-03-09T09:40:40.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:40.062+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c060a50 con 0x7f833802d7b0 2021-03-09T09:40:40.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:40:40 smithi146 bash[13136]: debug 2021-03-09T09:40:40.053+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11412 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:40:45.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.054+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:40:45.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.054+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:45.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.054+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4044 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c060d60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:45.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.054+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4044 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c060d60).stop 2021-03-09T09:40:45.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.054+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:45.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.054+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:45.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.054+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:45.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.054+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:45.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.054+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:45.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.054+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d1f0 con 0x7f8324003400 2021-03-09T09:40:45.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.054+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c230 con 0x7f8324008870 2021-03-09T09:40:45.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.054+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cac0 con 0x7f8324005e00 2021-03-09T09:40:45.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.058+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:40:45.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.058+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:45.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.058+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:40:45.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.058+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:45.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.058+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:45.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.058+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:45.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.058+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:45.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.058+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:45.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.058+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:45.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.058+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f8324008870 2021-03-09T09:40:45.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.058+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019f70 secure :-1 s=READY pgs=4052 cs=0 l=1 rev1=1 rx=0x7f833c02f550 tx=0x7f833c031000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:45.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.058+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c08f070 con 0x7f8324008870 2021-03-09T09:40:45.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.058+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f030 con 0x7f8324008870 2021-03-09T09:40:45.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:45.066+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8324008870 2021-03-09T09:40:45.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:40:45 smithi146 bash[13136]: debug 2021-03-09T09:40:45.057+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11437 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:40:50.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338019f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:40:50.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338019f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:50.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019f70 secure :-1 s=READY pgs=4052 cs=0 l=1 rev1=1 rx=0x7f833c02f550 tx=0x7f833c031000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:50.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019f70 secure :-1 s=READY pgs=4052 cs=0 l=1 rev1=1 rx=0x7f833c02f550 tx=0x7f833c031000).stop 2021-03-09T09:40:50.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338019f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:50.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019f70 unknown :-1 s=CLOSED pgs=4052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:50.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:50.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:50.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:50.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:40:50.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d1f0 con 0x7f8324003400 2021-03-09T09:40:50.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380107e0 con 0x7f833802d7b0 2021-03-09T09:40:50.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:50.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324005e00 2021-03-09T09:40:50.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:40:50.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:50.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:50.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:50.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:50.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:50.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:50.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.058+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:40:50.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.062+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4061 cs=0 l=1 rev1=1 rx=0x7f834404c690 tx=0x7f8344078300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:50.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.062+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405ce80 con 0x7f833802d7b0 2021-03-09T09:40:50.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.062+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833802d7b0 2021-03-09T09:40:50.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:50.066+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440530d0 con 0x7f833802d7b0 2021-03-09T09:40:50.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:40:50 smithi146 bash[13136]: debug 2021-03-09T09:40:50.057+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11467 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:40:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.058+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:40:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.058+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.058+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4061 cs=0 l=1 rev1=1 rx=0x7f834404c690 tx=0x7f8344078300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.058+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4061 cs=0 l=1 rev1=1 rx=0x7f834404c690 tx=0x7f8344078300).stop 2021-03-09T09:40:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:55.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:55.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:55.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c230 con 0x7f8324003400 2021-03-09T09:40:55.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bb80 con 0x7f8324008870 2021-03-09T09:40:55.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069f60 con 0x7f8324005e00 2021-03-09T09:40:55.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:40:55.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:40:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.062+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380248d0 con 0x7f8324008870 2021-03-09T09:40:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.066+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801da70 secure :-1 s=READY pgs=4070 cs=0 l=1 rev1=1 rx=0x7f833c079f00 tx=0x7f833c031000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.066+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c03f1b0 con 0x7f8324008870 2021-03-09T09:40:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.066+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f040 con 0x7f8324008870 2021-03-09T09:40:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:40:55.070+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8324008870 2021-03-09T09:40:55.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:40:55 smithi146 bash[13136]: debug 2021-03-09T09:40:55.061+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11489 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:41:00.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:41:00.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:00.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801da70 secure :-1 s=READY pgs=4070 cs=0 l=1 rev1=1 rx=0x7f833c079f00 tx=0x7f833c031000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:00.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801da70 secure :-1 s=READY pgs=4070 cs=0 l=1 rev1=1 rx=0x7f833c079f00 tx=0x7f833c031000).stop 2021-03-09T09:41:00.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801da70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:00.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801da70 unknown :-1 s=CLOSED pgs=4070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:00.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:00.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:00.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:41:00.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:00.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:00.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d1f0 con 0x7f8324003400 2021-03-09T09:41:00.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e270 con 0x7f833802d7b0 2021-03-09T09:41:00.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380107e0 con 0x7f8324005e00 2021-03-09T09:41:00.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:41:00.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:00.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.061+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:00.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.065+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:00.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.065+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:00.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.065+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:00.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.065+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:00.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.065+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f833802d7b0 2021-03-09T09:41:00.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.065+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4079 cs=0 l=1 rev1=1 rx=0x7f834400e240 tx=0x7f8344078300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:00.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.065+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440545f0 con 0x7f833802d7b0 2021-03-09T09:41:00.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.065+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833802d7b0 2021-03-09T09:41:00.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:00.073+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440791e0 con 0x7f833802d7b0 2021-03-09T09:41:00.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:41:00 smithi146 bash[13136]: debug 2021-03-09T09:41:00.065+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11513 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:41:05.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:41:05.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:05.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4079 cs=0 l=1 rev1=1 rx=0x7f834400e240 tx=0x7f8344078300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:05.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4079 cs=0 l=1 rev1=1 rx=0x7f834400e240 tx=0x7f8344078300).stop 2021-03-09T09:41:05.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:05.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:05.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:05.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:05.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:05.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380107e0 con 0x7f8324003400 2021-03-09T09:41:05.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324008870 2021-03-09T09:41:05.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019890 con 0x7f8324005e00 2021-03-09T09:41:05.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:41:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:41:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:05.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:05.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:05.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:05.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.065+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802c500 con 0x7f8324008870 2021-03-09T09:41:05.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.069+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801da70 secure :-1 s=READY pgs=4087 cs=0 l=1 rev1=1 rx=0x7f8338066f50 tx=0x7f8344078300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:05.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.069+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405cce0 con 0x7f8324008870 2021-03-09T09:41:05.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.069+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T09:41:05.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:05.069+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405b030 con 0x7f8324008870 2021-03-09T09:41:05.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:41:05 smithi146 bash[13136]: debug 2021-03-09T09:41:05.065+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11547 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:41:10.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.065+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:41:10.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.065+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:10.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.065+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801da70 secure :-1 s=READY pgs=4087 cs=0 l=1 rev1=1 rx=0x7f8338066f50 tx=0x7f8344078300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:10.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.065+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801da70 secure :-1 s=READY pgs=4087 cs=0 l=1 rev1=1 rx=0x7f8338066f50 tx=0x7f8344078300).stop 2021-03-09T09:41:10.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.065+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801da70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:10.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.065+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801da70 unknown :-1 s=CLOSED pgs=4087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:10.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.065+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:10.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.065+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:10.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.065+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:10.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.065+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d1f0 con 0x7f8324003400 2021-03-09T09:41:10.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.065+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380107e0 con 0x7f833802d7b0 2021-03-09T09:41:10.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.065+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802dd90 con 0x7f8324005e00 2021-03-09T09:41:10.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.065+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:41:10.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.065+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:10.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.065+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:41:10.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.065+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:10.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.069+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:10.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.069+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:10.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.069+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:10.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.069+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:10.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.069+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:10.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.069+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:41:10.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.069+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4096 cs=0 l=1 rev1=1 rx=0x7f83380428c0 tx=0x7f833c031000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:10.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.069+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c08f020 con 0x7f833802d7b0 2021-03-09T09:41:10.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.073+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f080 con 0x7f833802d7b0 2021-03-09T09:41:10.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:10.073+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c053270 con 0x7f833802d7b0 2021-03-09T09:41:10.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:41:10 smithi146 bash[13136]: debug 2021-03-09T09:41:10.068+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11565 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:41:15.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:41:15.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:15.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4096 cs=0 l=1 rev1=1 rx=0x7f83380428c0 tx=0x7f833c031000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:15.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4096 cs=0 l=1 rev1=1 rx=0x7f83380428c0 tx=0x7f833c031000).stop 2021-03-09T09:41:15.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:15.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:15.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:15.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338070330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:15.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:15.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380107e0 con 0x7f8324003400 2021-03-09T09:41:15.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002a00 con 0x7f8324008870 2021-03-09T09:41:15.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020430 con 0x7f8324005e00 2021-03-09T09:41:15.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:41:15.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:15.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:41:15.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:15.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.069+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338070330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:15.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.073+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:15.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.073+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:15.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.073+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:15.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.073+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:15.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.073+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380098d0 con 0x7f8324008870 2021-03-09T09:41:15.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.073+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338070330 secure :-1 s=READY pgs=4104 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344078300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:15.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.073+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344071dd0 con 0x7f8324008870 2021-03-09T09:41:15.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.073+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T09:41:15.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:15.077+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344079c30 con 0x7f8324008870 2021-03-09T09:41:15.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:41:15 smithi146 bash[13136]: debug 2021-03-09T09:41:15.068+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11591 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:41:20.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:41:20.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:20.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338070330 secure :-1 s=READY pgs=4104 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344078300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:20.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338070330 secure :-1 s=READY pgs=4104 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344078300).stop 2021-03-09T09:41:20.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338070330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:20.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338070330 unknown :-1 s=CLOSED pgs=4104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:20.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:20.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:20.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:20.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380107e0 con 0x7f8324003400 2021-03-09T09:41:20.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002a00 con 0x7f833802d7b0 2021-03-09T09:41:20.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380614d0 con 0x7f8324005e00 2021-03-09T09:41:20.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:41:20.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:20.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:41:20.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:20.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.073+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:20.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.077+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:20.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.077+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:20.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.077+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:20.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.077+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:20.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.077+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020ba0 con 0x7f833802d7b0 2021-03-09T09:41:20.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.077+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4116 cs=0 l=1 rev1=1 rx=0x7f8344067000 tx=0x7f83440133f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:20.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.077+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400f6b0 con 0x7f833802d7b0 2021-03-09T09:41:20.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.077+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344071dd0 con 0x7f833802d7b0 2021-03-09T09:41:20.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:20.081+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344010ca0 con 0x7f833802d7b0 2021-03-09T09:41:20.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:41:20 smithi146 bash[13136]: debug 2021-03-09T09:41:20.072+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11621 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:41:25.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:41:25.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:25.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4116 cs=0 l=1 rev1=1 rx=0x7f8344067000 tx=0x7f83440133f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:25.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4116 cs=0 l=1 rev1=1 rx=0x7f8344067000 tx=0x7f83440133f0).stop 2021-03-09T09:41:25.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:25.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:25.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:25.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:25.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:25.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380107e0 con 0x7f8324003400 2021-03-09T09:41:25.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028e90 con 0x7f8324008870 2021-03-09T09:41:25.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e110 con 0x7f8324005e00 2021-03-09T09:41:25.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:41:25.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:25.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:41:25.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:25.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:25.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:25.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:25.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:25.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:25.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f8324008870 2021-03-09T09:41:25.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016730 secure :-1 s=READY pgs=4123 cs=0 l=1 rev1=1 rx=0x7f8338066f50 tx=0x7f833c031000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:25.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c08f040 con 0x7f8324008870 2021-03-09T09:41:25.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.077+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8324008870 2021-03-09T09:41:25.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:25.081+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f8324008870 2021-03-09T09:41:25.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:41:25 smithi146 bash[13136]: debug 2021-03-09T09:41:25.076+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11641 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:41:30.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338016730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:41:30.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338016730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:30.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016730 secure :-1 s=READY pgs=4123 cs=0 l=1 rev1=1 rx=0x7f8338066f50 tx=0x7f833c031000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:30.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016730 secure :-1 s=READY pgs=4123 cs=0 l=1 rev1=1 rx=0x7f8338066f50 tx=0x7f833c031000).stop 2021-03-09T09:41:30.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338016730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:30.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016730 unknown :-1 s=CLOSED pgs=4123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:30.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:30.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:30.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:30.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002a00 con 0x7f8324003400 2021-03-09T09:41:30.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028e90 con 0x7f833802d7b0 2021-03-09T09:41:30.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806fd20 con 0x7f8324005e00 2021-03-09T09:41:30.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:41:30.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:30.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:41:30.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:30.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:30.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:30.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:30.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:30.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:30.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.077+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:41:30.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.081+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4130 cs=0 l=1 rev1=1 rx=0x7f8344013510 tx=0x7f83440133f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:30.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.081+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834407ada0 con 0x7f833802d7b0 2021-03-09T09:41:30.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.081+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833802d7b0 2021-03-09T09:41:30.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:30.085+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344072d80 con 0x7f833802d7b0 2021-03-09T09:41:30.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:41:30 smithi146 bash[13136]: debug 2021-03-09T09:41:30.076+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11669 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:41:35.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:41:35.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:35.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4130 cs=0 l=1 rev1=1 rx=0x7f8344013510 tx=0x7f83440133f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:35.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4130 cs=0 l=1 rev1=1 rx=0x7f8344013510 tx=0x7f83440133f0).stop 2021-03-09T09:41:35.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:35.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:35.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:35.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:35.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:35.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024bf0 con 0x7f8324003400 2021-03-09T09:41:35.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:35.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:41:35.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:35.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:41:35.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:35.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380157e0 con 0x7f8324008870 2021-03-09T09:41:35.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380708a0 con 0x7f8324005e00 2021-03-09T09:41:35.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:35.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:35.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:35.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:35.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017e40 con 0x7f8324008870 2021-03-09T09:41:35.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007700 secure :-1 s=READY pgs=4139 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f83440133f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:35.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344016300 con 0x7f8324008870 2021-03-09T09:41:35.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.081+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033040 con 0x7f8324008870 2021-03-09T09:41:35.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:35.085+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8324008870 2021-03-09T09:41:35.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:41:35 smithi146 bash[13136]: debug 2021-03-09T09:41:35.080+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11700 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:41:40.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.085+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:41:40.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.085+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:40.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.085+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007700 secure :-1 s=READY pgs=4139 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f83440133f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:40.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.085+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007700 secure :-1 s=READY pgs=4139 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f83440133f0).stop 2021-03-09T09:41:40.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.085+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:40.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.085+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007700 unknown :-1 s=CLOSED pgs=4139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:40.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.085+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:40.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.085+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:40.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.085+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:40.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.085+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e270 con 0x7f8324003400 2021-03-09T09:41:40.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.085+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028e90 con 0x7f833802d7b0 2021-03-09T09:41:40.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.085+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380708a0 con 0x7f8324005e00 2021-03-09T09:41:40.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.085+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:41:40.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.085+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:40.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.085+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:41:40.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.089+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:40.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.089+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:40.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.089+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:40.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.089+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:40.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.089+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:40.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.089+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:40.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.089+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068da0 con 0x7f833802d7b0 2021-03-09T09:41:40.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.093+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4151 cs=0 l=1 rev1=1 rx=0x7f8344007ce0 tx=0x7f83440133f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:40.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.093+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344078190 con 0x7f833802d7b0 2021-03-09T09:41:40.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.093+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833802d7b0 2021-03-09T09:41:40.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:40.093+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833802d7b0 2021-03-09T09:41:40.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:41:40 smithi146 bash[13136]: debug 2021-03-09T09:41:40.080+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11716 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:41:45.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:41:45.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:45.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4151 cs=0 l=1 rev1=1 rx=0x7f8344007ce0 tx=0x7f83440133f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:45.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4151 cs=0 l=1 rev1=1 rx=0x7f8344007ce0 tx=0x7f83440133f0).stop 2021-03-09T09:41:45.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:45.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:45.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:45.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:45.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:45.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802dd90 con 0x7f8324003400 2021-03-09T09:41:45.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009dc0 con 0x7f8324008870 2021-03-09T09:41:45.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028e90 con 0x7f8324005e00 2021-03-09T09:41:45.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:41:45.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:45.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:41:45.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:45.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:45.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:45.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:45.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:45.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:45.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.084+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380104c0 con 0x7f8324008870 2021-03-09T09:41:45.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.088+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027740 secure :-1 s=READY pgs=4156 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f83440133f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:45.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344014c70 con 0x7f8324008870 2021-03-09T09:41:45.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033050 con 0x7f8324008870 2021-03-09T09:41:45.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:45.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834401fba0 con 0x7f8324008870 2021-03-09T09:41:45.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:41:45 smithi146 bash[13136]: debug 2021-03-09T09:41:45.084+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11743 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:41:50.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:41:50.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:50.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027740 secure :-1 s=READY pgs=4156 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f83440133f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:50.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027740 secure :-1 s=READY pgs=4156 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f83440133f0).stop 2021-03-09T09:41:50.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:50.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027740 unknown :-1 s=CLOSED pgs=4156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:50.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:50.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:50.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:50.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025010 con 0x7f8324003400 2021-03-09T09:41:50.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028e90 con 0x7f833802d7b0 2021-03-09T09:41:50.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380157e0 con 0x7f8324005e00 2021-03-09T09:41:50.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:41:50.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:50.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:41:50.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:50.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:50.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:50.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:50.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:50.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:50.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f833802d7b0 2021-03-09T09:41:50.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.088+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4169 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c031000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:50.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.092+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c08f020 con 0x7f833802d7b0 2021-03-09T09:41:50.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.092+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f030 con 0x7f833802d7b0 2021-03-09T09:41:50.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:50.092+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833802d7b0 2021-03-09T09:41:50.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:41:50 smithi146 bash[13136]: debug 2021-03-09T09:41:50.084+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11767 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:41:55.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:41:55.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:55.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4169 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c031000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:55.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4169 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c031000).stop 2021-03-09T09:41:55.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:55.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:55.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:55.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:55.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380083a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:55.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c9e0 con 0x7f8324003400 2021-03-09T09:41:55.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324008870 2021-03-09T09:41:55.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020440 con 0x7f8324005e00 2021-03-09T09:41:55.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:41:55.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:55.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:41:55.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.088+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380083a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.092+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.092+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.092+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.092+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:55.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.092+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f8324008870 2021-03-09T09:41:55.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.092+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380083a0 secure :-1 s=READY pgs=4178 cs=0 l=1 rev1=1 rx=0x7f833803a060 tx=0x7f833c040320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:55.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.092+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c08f070 con 0x7f8324008870 2021-03-09T09:41:55.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.092+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f030 con 0x7f8324008870 2021-03-09T09:41:55.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:41:55.096+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f8324008870 2021-03-09T09:41:55.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:41:55 smithi146 bash[13136]: debug 2021-03-09T09:41:55.088+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11802 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:42:00.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.088+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380083a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:42:00.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.088+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380083a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:00.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.088+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380083a0 secure :-1 s=READY pgs=4178 cs=0 l=1 rev1=1 rx=0x7f833803a060 tx=0x7f833c040320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:00.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.088+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380083a0 secure :-1 s=READY pgs=4178 cs=0 l=1 rev1=1 rx=0x7f833803a060 tx=0x7f833c040320).stop 2021-03-09T09:42:00.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380083a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:00.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.088+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380083a0 unknown :-1 s=CLOSED pgs=4178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:00.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.088+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:00.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.088+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:00.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.088+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:00.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a30 con 0x7f8324003400 2021-03-09T09:42:00.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009dc0 con 0x7f833802d7b0 2021-03-09T09:42:00.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028e90 con 0x7f8324005e00 2021-03-09T09:42:00.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.088+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:42:00.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.088+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:00.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.088+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:42:00.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.088+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:00.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.092+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:00.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.092+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:00.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.092+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:00.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.092+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:00.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.092+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:00.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.092+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028b50 con 0x7f833802d7b0 2021-03-09T09:42:00.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.092+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4182 cs=0 l=1 rev1=1 rx=0x7f833c0787f0 tx=0x7f833c040320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:00.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.092+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c08f070 con 0x7f833802d7b0 2021-03-09T09:42:00.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.092+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f030 con 0x7f833802d7b0 2021-03-09T09:42:00.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:00.096+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f833802d7b0 2021-03-09T09:42:00.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:42:00 smithi146 bash[13136]: debug 2021-03-09T09:42:00.092+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11819 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:42:05.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:42:05.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:05.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4182 cs=0 l=1 rev1=1 rx=0x7f833c0787f0 tx=0x7f833c040320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:05.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4182 cs=0 l=1 rev1=1 rx=0x7f833c0787f0 tx=0x7f833c040320).stop 2021-03-09T09:42:05.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:05.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:05.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:05.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380141d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:05.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:05.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c9e0 con 0x7f8324003400 2021-03-09T09:42:05.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019110 con 0x7f8324008870 2021-03-09T09:42:05.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028e90 con 0x7f8324005e00 2021-03-09T09:42:05.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:42:05.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:05.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:42:05.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:05.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.092+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:05.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.096+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:05.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.096+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:05.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.096+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:05.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.096+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:05.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.096+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020ba0 con 0x7f8324008870 2021-03-09T09:42:05.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.096+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380141d0 secure :-1 s=READY pgs=4191 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f83440133f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:05.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.096+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404afe0 con 0x7f8324008870 2021-03-09T09:42:05.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.096+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T09:42:05.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:05.100+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344078190 con 0x7f8324008870 2021-03-09T09:42:05.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:42:05 smithi146 bash[13136]: debug 2021-03-09T09:42:05.096+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11849 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:42:10.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.092+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380141d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:42:10.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.092+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380141d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:10.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.092+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380141d0 secure :-1 s=READY pgs=4191 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f83440133f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.092+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380141d0 secure :-1 s=READY pgs=4191 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f83440133f0).stop 2021-03-09T09:42:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.092+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380141d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.092+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380141d0 unknown :-1 s=CLOSED pgs=4191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.092+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.092+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.092+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.092+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324003400 2021-03-09T09:42:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.092+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062340 con 0x7f833802d7b0 2021-03-09T09:42:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.092+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022da0 con 0x7f8324005e00 2021-03-09T09:42:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.096+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:42:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.096+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.096+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:42:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.096+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.096+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.096+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:10.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.096+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:10.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.096+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:10.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.096+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:10.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.096+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380185e0 con 0x7f833802d7b0 2021-03-09T09:42:10.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.096+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4200 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c040320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:10.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.100+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c08f030 con 0x7f833802d7b0 2021-03-09T09:42:10.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.100+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:42:10.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:10.104+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055310 con 0x7f833802d7b0 2021-03-09T09:42:10.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:42:10 smithi146 bash[13136]: debug 2021-03-09T09:42:10.100+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11869 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:42:15.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:42:15.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:15.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4200 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c040320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:15.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4200 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c040320).stop 2021-03-09T09:42:15.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:15.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:15.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380141d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:15.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:15.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:15.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a350 con 0x7f8324003400 2021-03-09T09:42:15.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060c40 con 0x7f8324008870 2021-03-09T09:42:15.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f050 con 0x7f8324005e00 2021-03-09T09:42:15.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004c00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:42:15.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:42:15.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:15.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:15.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:15.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:15.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:15.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:15.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:15.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f8324008870 2021-03-09T09:42:15.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380141d0 secure :-1 s=READY pgs=4208 cs=0 l=1 rev1=1 rx=0x7f8338066f80 tx=0x7f833c040320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:15.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c08f070 con 0x7f8324008870 2021-03-09T09:42:15.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.100+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0298e0 con 0x7f8324008870 2021-03-09T09:42:15.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:15.104+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06a2d0 con 0x7f8324008870 2021-03-09T09:42:15.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:42:15 smithi146 bash[13136]: debug 2021-03-09T09:42:15.100+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11896 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:42:20.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.096+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380141d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:42:20.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.096+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380141d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:20.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.096+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380141d0 secure :-1 s=READY pgs=4208 cs=0 l=1 rev1=1 rx=0x7f8338066f80 tx=0x7f833c040320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:20.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380141d0 secure :-1 s=READY pgs=4208 cs=0 l=1 rev1=1 rx=0x7f8338066f80 tx=0x7f833c040320).stop 2021-03-09T09:42:20.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380141d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:20.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380141d0 unknown :-1 s=CLOSED pgs=4208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:20.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:20.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:20.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:20.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009dc0 con 0x7f8324003400 2021-03-09T09:42:20.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028e90 con 0x7f833802d7b0 2021-03-09T09:42:20.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e830 con 0x7f8324005e00 2021-03-09T09:42:20.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:42:20.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:20.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:42:20.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:20.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:20.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:20.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:20.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:20.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:20.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a5a0 con 0x7f833802d7b0 2021-03-09T09:42:20.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.100+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4217 cs=0 l=1 rev1=1 rx=0x7f8344072db0 tx=0x7f83440133f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:20.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.104+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344017e00 con 0x7f833802d7b0 2021-03-09T09:42:20.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.104+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072bb0 con 0x7f833802d7b0 2021-03-09T09:42:20.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:20.108+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344049710 con 0x7f833802d7b0 2021-03-09T09:42:20.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:42:20 smithi146 bash[13136]: debug 2021-03-09T09:42:20.104+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11920 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:42:25.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:42:25.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:25.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4217 cs=0 l=1 rev1=1 rx=0x7f8344072db0 tx=0x7f83440133f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:25.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4217 cs=0 l=1 rev1=1 rx=0x7f8344072db0 tx=0x7f83440133f0).stop 2021-03-09T09:42:25.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:25.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:25.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:25.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:25.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:25.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005e60 con 0x7f8324003400 2021-03-09T09:42:25.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806fd20 con 0x7f8324008870 2021-03-09T09:42:25.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c9e0 con 0x7f8324005e00 2021-03-09T09:42:25.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:42:25.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:25.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:42:25.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:25.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:25.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:25.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:25.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:25.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:25.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f8324008870 2021-03-09T09:42:25.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.104+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380140f0 secure :-1 s=READY pgs=4227 cs=0 l=1 rev1=1 rx=0x7f833c071eb0 tx=0x7f833c040320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:25.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.108+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0794d0 con 0x7f8324008870 2021-03-09T09:42:25.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.112+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8324008870 2021-03-09T09:42:25.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:25.112+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8324008870 2021-03-09T09:42:25.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:42:25 smithi146 bash[13136]: debug 2021-03-09T09:42:25.103+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11947 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:42:30.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380140f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:42:30.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380140f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:30.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380140f0 secure :-1 s=READY pgs=4227 cs=0 l=1 rev1=1 rx=0x7f833c071eb0 tx=0x7f833c040320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:30.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380140f0 secure :-1 s=READY pgs=4227 cs=0 l=1 rev1=1 rx=0x7f833c071eb0 tx=0x7f833c040320).stop 2021-03-09T09:42:30.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380140f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:30.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380140f0 unknown :-1 s=CLOSED pgs=4227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:30.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:30.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:30.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:30.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022da0 con 0x7f8324003400 2021-03-09T09:42:30.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f9b0 con 0x7f833802d7b0 2021-03-09T09:42:30.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025830 con 0x7f8324005e00 2021-03-09T09:42:30.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:42:30.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:30.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:42:30.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:30.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:30.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:30.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:30.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:30.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:30.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022060 con 0x7f833802d7b0 2021-03-09T09:42:30.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4234 cs=0 l=1 rev1=1 rx=0x7f833c069ad0 tx=0x7f833c040320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:30.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c08f070 con 0x7f833802d7b0 2021-03-09T09:42:30.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.107+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:42:30.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:30.111+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06a2d0 con 0x7f833802d7b0 2021-03-09T09:42:30.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:42:30 smithi146 bash[13136]: debug 2021-03-09T09:42:30.107+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11971 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:42:35.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.107+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:42:35.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.107+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:35.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.107+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4234 cs=0 l=1 rev1=1 rx=0x7f833c069ad0 tx=0x7f833c040320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:35.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.107+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4234 cs=0 l=1 rev1=1 rx=0x7f833c069ad0 tx=0x7f833c040320).stop 2021-03-09T09:42:35.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.107+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:35.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.107+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:35.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.107+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:35.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.107+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:35.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.107+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:35.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.107+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061130 con 0x7f8324003400 2021-03-09T09:42:35.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.107+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007290 con 0x7f8324008870 2021-03-09T09:42:35.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.107+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:42:35.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.107+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e830 con 0x7f8324005e00 2021-03-09T09:42:35.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.107+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:35.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.107+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:42:35.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.107+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:35.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.111+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:35.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.111+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:35.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.111+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:35.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.111+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:35.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.111+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:35.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.111+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e8d0 con 0x7f8324008870 2021-03-09T09:42:35.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.111+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380140f0 secure :-1 s=READY pgs=4245 cs=0 l=1 rev1=1 rx=0x7f833c018960 tx=0x7f833c046040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:35.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.115+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c060470 con 0x7f8324008870 2021-03-09T09:42:35.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.115+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c060870 con 0x7f8324008870 2021-03-09T09:42:35.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:35.119+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06a370 con 0x7f8324008870 2021-03-09T09:42:35.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:42:35 smithi146 bash[13136]: debug 2021-03-09T09:42:35.111+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 11998 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:42:40.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.111+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380140f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:42:40.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.111+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380140f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:40.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.111+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380140f0 secure :-1 s=READY pgs=4245 cs=0 l=1 rev1=1 rx=0x7f833c018960 tx=0x7f833c046040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:40.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.111+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380140f0 secure :-1 s=READY pgs=4245 cs=0 l=1 rev1=1 rx=0x7f833c018960 tx=0x7f833c046040).stop 2021-03-09T09:42:40.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.111+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380140f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:40.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.111+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380140f0 unknown :-1 s=CLOSED pgs=4245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:40.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.111+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:40.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.111+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:40.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.111+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:40.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.111+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e5b0 con 0x7f8324003400 2021-03-09T09:42:40.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.111+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f833802d7b0 2021-03-09T09:42:40.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.111+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025830 con 0x7f8324005e00 2021-03-09T09:42:40.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.111+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:42:40.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.111+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:40.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.111+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:42:40.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.111+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:40.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:40.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:40.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:40.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:40.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:40.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b5f0 con 0x7f833802d7b0 2021-03-09T09:42:40.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4254 cs=0 l=1 rev1=1 rx=0x7f833c05ff40 tx=0x7f833c046040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:40.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.115+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c06a520 con 0x7f833802d7b0 2021-03-09T09:42:40.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.115+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0187c0 con 0x7f833802d7b0 2021-03-09T09:42:40.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:40.119+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06a310 con 0x7f833802d7b0 2021-03-09T09:42:40.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:42:40 smithi146 bash[13136]: debug 2021-03-09T09:42:40.111+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12023 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:42:45.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:42:45.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:45.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4254 cs=0 l=1 rev1=1 rx=0x7f833c05ff40 tx=0x7f833c046040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:45.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4254 cs=0 l=1 rev1=1 rx=0x7f833c05ff40 tx=0x7f833c046040).stop 2021-03-09T09:42:45.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:45.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:45.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:45.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:45.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:45.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c9e0 con 0x7f8324003400 2021-03-09T09:42:45.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061130 con 0x7f8324008870 2021-03-09T09:42:45.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060e40 con 0x7f8324005e00 2021-03-09T09:42:45.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:42:45.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:42:45.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:45.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:45.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:45.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:45.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:45.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:45.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:45.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f8324008870 2021-03-09T09:42:45.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.119+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027720 secure :-1 s=READY pgs=4260 cs=0 l=1 rev1=1 rx=0x7f833c078fe0 tx=0x7f833c018720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:45.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.119+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c060870 con 0x7f8324008870 2021-03-09T09:42:45.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.119+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8324008870 2021-03-09T09:42:45.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:45.123+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0715b0 con 0x7f8324008870 2021-03-09T09:42:45.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:42:45 smithi146 bash[13136]: debug 2021-03-09T09:42:45.115+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12048 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:42:50.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:42:50.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:50.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027720 secure :-1 s=READY pgs=4260 cs=0 l=1 rev1=1 rx=0x7f833c078fe0 tx=0x7f833c018720).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:50.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027720 secure :-1 s=READY pgs=4260 cs=0 l=1 rev1=1 rx=0x7f833c078fe0 tx=0x7f833c018720).stop 2021-03-09T09:42:50.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:50.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027720 unknown :-1 s=CLOSED pgs=4260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:50.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:50.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:50.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:50.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d880 con 0x7f8324003400 2021-03-09T09:42:50.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380248d0 con 0x7f833802d7b0 2021-03-09T09:42:50.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026720 con 0x7f8324005e00 2021-03-09T09:42:50.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:42:50.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:50.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:42:50.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:50.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:50.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.119+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:50.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.119+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:50.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.119+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:50.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.119+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:50.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.119+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380185e0 con 0x7f833802d7b0 2021-03-09T09:42:50.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.119+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4269 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c018720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:50.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.119+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c08f030 con 0x7f833802d7b0 2021-03-09T09:42:50.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.119+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:42:50.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:50.127+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c061a10 con 0x7f833802d7b0 2021-03-09T09:42:50.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:42:50 smithi146 bash[13136]: debug 2021-03-09T09:42:50.115+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12077 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:42:55.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:42:55.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:55.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4269 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c018720).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:55.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4269 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c018720).stop 2021-03-09T09:42:55.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:55.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:55.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380202e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:55.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:55.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:55.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e830 con 0x7f8324003400 2021-03-09T09:42:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007290 con 0x7f8324008870 2021-03-09T09:42:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061130 con 0x7f8324005e00 2021-03-09T09:42:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:42:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:42:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.119+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380202e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.123+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.123+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.123+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.123+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:55.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.123+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f8324008870 2021-03-09T09:42:55.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.123+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380202e0 secure :-1 s=READY pgs=4278 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c018720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:55.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.123+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c08f070 con 0x7f8324008870 2021-03-09T09:42:55.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.123+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8324008870 2021-03-09T09:42:55.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:42:55.127+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c061a10 con 0x7f8324008870 2021-03-09T09:42:55.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:42:55 smithi146 bash[13136]: debug 2021-03-09T09:42:55.119+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12099 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:43:00.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.119+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380202e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:43:00.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.119+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380202e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:00.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.119+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380202e0 secure :-1 s=READY pgs=4278 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c018720).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:00.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.119+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380202e0 secure :-1 s=READY pgs=4278 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c018720).stop 2021-03-09T09:43:00.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.119+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380202e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:00.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.119+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380202e0 unknown :-1 s=CLOSED pgs=4278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:00.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.119+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:00.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.119+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:00.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.119+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:00.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.119+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380664d0 con 0x7f8324003400 2021-03-09T09:43:00.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.119+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f833802d7b0 2021-03-09T09:43:00.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.119+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023dd0 con 0x7f8324005e00 2021-03-09T09:43:00.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.119+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:43:00.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.119+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:00.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.119+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:43:00.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.119+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:00.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.123+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:00.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.123+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:00.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.123+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:00.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.123+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:00.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.123+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:00.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.123+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380116f0 con 0x7f833802d7b0 2021-03-09T09:43:00.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.123+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4286 cs=0 l=1 rev1=1 rx=0x7f834405f040 tx=0x7f83440133f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:00.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.123+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344003e50 con 0x7f833802d7b0 2021-03-09T09:43:00.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.123+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404aa60 con 0x7f833802d7b0 2021-03-09T09:43:00.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:00.127+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404c8f0 con 0x7f833802d7b0 2021-03-09T09:43:00.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:43:00 smithi146 bash[13136]: debug 2021-03-09T09:43:00.123+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12124 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:43:05.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.123+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:43:05.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.123+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:05.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.123+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4286 cs=0 l=1 rev1=1 rx=0x7f834405f040 tx=0x7f83440133f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:05.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.123+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4286 cs=0 l=1 rev1=1 rx=0x7f834405f040 tx=0x7f83440133f0).stop 2021-03-09T09:43:05.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.123+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:05.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.123+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:05.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.123+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:05.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.123+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:43:05.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.123+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:05.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.123+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:05.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.123+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:05.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.123+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061130 con 0x7f8324003400 2021-03-09T09:43:05.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.123+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:43:05.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.123+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:05.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.123+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d880 con 0x7f8324008870 2021-03-09T09:43:05.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.123+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380664d0 con 0x7f8324005e00 2021-03-09T09:43:05.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.127+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:05.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.127+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:05.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.127+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:05.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.127+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:05.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.127+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:05.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.127+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802c9e0 con 0x7f8324008870 2021-03-09T09:43:05.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.127+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 secure :-1 s=READY pgs=4298 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f83440133f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:05.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.131+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404b6f0 con 0x7f8324008870 2021-03-09T09:43:05.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.131+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T09:43:05.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:05.135+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404b6f0 con 0x7f8324008870 2021-03-09T09:43:05.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:43:05 smithi146 bash[13136]: debug 2021-03-09T09:43:05.123+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12150 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:43:10.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:43:10.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:10.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 secure :-1 s=READY pgs=4298 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f83440133f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:10.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 secure :-1 s=READY pgs=4298 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f83440133f0).stop 2021-03-09T09:43:10.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:10.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 unknown :-1 s=CLOSED pgs=4298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:10.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:10.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:10.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:10.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023dd0 con 0x7f8324003400 2021-03-09T09:43:10.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806ff20 con 0x7f833802d7b0 2021-03-09T09:43:10.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007290 con 0x7f8324005e00 2021-03-09T09:43:10.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:43:10.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:10.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:43:10.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:10.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.127+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:10.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.131+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:10.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.131+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:10.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.131+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:10.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.131+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:10.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.131+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d60 con 0x7f833802d7b0 2021-03-09T09:43:10.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.131+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4306 cs=0 l=1 rev1=1 rx=0x7f833c01cf20 tx=0x7f833c018720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:10.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.131+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c08f070 con 0x7f833802d7b0 2021-03-09T09:43:10.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.131+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:43:10.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:10.135+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055310 con 0x7f833802d7b0 2021-03-09T09:43:10.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:43:10 smithi146 bash[13136]: debug 2021-03-09T09:43:10.127+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12179 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:43:15.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.126+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:43:15.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.126+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:15.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.126+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4306 cs=0 l=1 rev1=1 rx=0x7f833c01cf20 tx=0x7f833c018720).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:15.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.126+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4306 cs=0 l=1 rev1=1 rx=0x7f833c01cf20 tx=0x7f833c018720).stop 2021-03-09T09:43:15.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.126+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:15.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.126+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:15.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.126+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:15.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.126+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:15.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.126+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:15.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.126+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061130 con 0x7f8324003400 2021-03-09T09:43:15.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.126+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324008870 2021-03-09T09:43:15.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.126+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380127c0 con 0x7f8324005e00 2021-03-09T09:43:15.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.130+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:43:15.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.130+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:15.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.130+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:43:15.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.130+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:15.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.130+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:15.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.130+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:15.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.130+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:15.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.130+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:15.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.130+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:15.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.130+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f8324008870 2021-03-09T09:43:15.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.134+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 secure :-1 s=READY pgs=4314 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f83440133f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:15.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.134+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344079ea0 con 0x7f8324008870 2021-03-09T09:43:15.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.134+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T09:43:15.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:15.138+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8324008870 2021-03-09T09:43:15.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:43:15 smithi146 bash[13136]: debug 2021-03-09T09:43:15.131+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12204 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:43:20.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:43:20.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:20.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 secure :-1 s=READY pgs=4314 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f83440133f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:20.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 secure :-1 s=READY pgs=4314 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f83440133f0).stop 2021-03-09T09:43:20.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:20.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 unknown :-1 s=CLOSED pgs=4314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:20.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:20.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:20.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:20.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f9b0 con 0x7f8324003400 2021-03-09T09:43:20.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e830 con 0x7f833802d7b0 2021-03-09T09:43:20.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a350 con 0x7f8324005e00 2021-03-09T09:43:20.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:43:20.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:20.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:20.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:43:20.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:20.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:20.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:20.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:20.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:20.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.130+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:43:20.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.134+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4321 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f83440133f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:20.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.134+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344069400 con 0x7f833802d7b0 2021-03-09T09:43:20.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.134+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344069560 con 0x7f833802d7b0 2021-03-09T09:43:20.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:20.138+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f833802d7b0 2021-03-09T09:43:20.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:43:20 smithi146 bash[13136]: debug 2021-03-09T09:43:20.131+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12233 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:43:25.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.130+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:43:25.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.130+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:25.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.130+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4321 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f83440133f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:25.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.130+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4321 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f83440133f0).stop 2021-03-09T09:43:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:25.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380175a0 con 0x7f8324003400 2021-03-09T09:43:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007290 con 0x7f8324008870 2021-03-09T09:43:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061130 con 0x7f8324005e00 2021-03-09T09:43:25.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:43:25.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:25.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:43:25.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:25.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.134+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338009be0 con 0x7f8324008870 2021-03-09T09:43:25.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.138+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 secure :-1 s=READY pgs=4331 cs=0 l=1 rev1=1 rx=0x7f833806c4d0 tx=0x7f833c018720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:25.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.138+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00f9e0 con 0x7f8324008870 2021-03-09T09:43:25.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.138+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8324008870 2021-03-09T09:43:25.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:25.142+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8324008870 2021-03-09T09:43:25.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:43:25 smithi146 bash[13136]: debug 2021-03-09T09:43:25.135+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12255 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:43:30.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.134+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:43:30.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.134+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:30.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.134+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 secure :-1 s=READY pgs=4331 cs=0 l=1 rev1=1 rx=0x7f833806c4d0 tx=0x7f833c018720).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:30.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.134+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 secure :-1 s=READY pgs=4331 cs=0 l=1 rev1=1 rx=0x7f833806c4d0 tx=0x7f833c018720).stop 2021-03-09T09:43:30.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.134+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:30.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.134+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f20 unknown :-1 s=CLOSED pgs=4331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:30.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.134+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:30.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.134+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:30.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.134+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:30.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.134+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065d20 con 0x7f8324003400 2021-03-09T09:43:30.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.134+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e830 con 0x7f833802d7b0 2021-03-09T09:43:30.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.134+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a350 con 0x7f8324005e00 2021-03-09T09:43:30.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.134+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:43:30.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.134+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:30.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.134+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:43:30.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.134+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:30.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.138+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:30.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.138+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:30.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.138+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:30.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.138+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:30.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.138+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:30.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.138+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:43:30.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.138+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4338 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c018720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:30.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.138+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00c360 con 0x7f833802d7b0 2021-03-09T09:43:30.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.138+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01ba80 con 0x7f833802d7b0 2021-03-09T09:43:30.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:30.142+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c069da0 con 0x7f833802d7b0 2021-03-09T09:43:30.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:43:30 smithi146 bash[13136]: debug 2021-03-09T09:43:30.138+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12285 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:43:35.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:43:35.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:35.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4338 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c018720).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:35.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4338 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c018720).stop 2021-03-09T09:43:35.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:35.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:35.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338028600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:35.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:35.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:35.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f050 con 0x7f8324003400 2021-03-09T09:43:35.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061130 con 0x7f8324008870 2021-03-09T09:43:35.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a350 con 0x7f8324005e00 2021-03-09T09:43:35.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:43:35.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:43:35.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:35.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:35.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338028600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:35.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:35.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:35.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:35.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:35.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.138+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005910 con 0x7f8324008870 2021-03-09T09:43:35.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.142+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338028600 secure :-1 s=READY pgs=4347 cs=0 l=1 rev1=1 rx=0x7f833c0787f0 tx=0x7f833c065000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:35.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.142+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01a270 con 0x7f8324008870 2021-03-09T09:43:35.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.142+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01a270 con 0x7f8324008870 2021-03-09T09:43:35.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:35.146+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054720 con 0x7f8324008870 2021-03-09T09:43:35.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:43:35 smithi146 bash[13136]: debug 2021-03-09T09:43:35.138+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12303 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:43:40.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.138+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338028600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:43:40.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.138+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338028600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:40.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.138+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338028600 secure :-1 s=READY pgs=4347 cs=0 l=1 rev1=1 rx=0x7f833c0787f0 tx=0x7f833c065000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:40.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.138+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338028600 secure :-1 s=READY pgs=4347 cs=0 l=1 rev1=1 rx=0x7f833c0787f0 tx=0x7f833c065000).stop 2021-03-09T09:43:40.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.138+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338028600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:40.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.138+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338028600 unknown :-1 s=CLOSED pgs=4347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:40.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.138+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:40.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.138+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:40.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.138+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:40.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.138+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d10 con 0x7f8324003400 2021-03-09T09:43:40.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.138+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007290 con 0x7f833802d7b0 2021-03-09T09:43:40.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.138+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f050 con 0x7f8324005e00 2021-03-09T09:43:40.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.142+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:43:40.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.142+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:40.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.142+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:43:40.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.142+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:40.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.146+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:40.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.146+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:40.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.146+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:40.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.146+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:40.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.146+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:40.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.146+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:43:40.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.150+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4357 cs=0 l=1 rev1=1 rx=0x7f833c019010 tx=0x7f833c065000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:40.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.150+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c08f070 con 0x7f833802d7b0 2021-03-09T09:43:40.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.150+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:43:40.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:40.150+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c078220 con 0x7f833802d7b0 2021-03-09T09:43:40.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:43:40 smithi146 bash[13136]: debug 2021-03-09T09:43:40.142+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12334 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:43:45.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.142+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:43:45.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.142+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:45.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.142+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4357 cs=0 l=1 rev1=1 rx=0x7f833c019010 tx=0x7f833c065000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:45.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.142+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4357 cs=0 l=1 rev1=1 rx=0x7f833c019010 tx=0x7f833c065000).stop 2021-03-09T09:43:45.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.142+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:45.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.142+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:45.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.142+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:45.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.142+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338067ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:45.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.142+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:45.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.142+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f8324003400 2021-03-09T09:43:45.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.142+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324008870 2021-03-09T09:43:45.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.142+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380068a0 con 0x7f8324005e00 2021-03-09T09:43:45.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.142+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:43:45.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.142+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:45.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.142+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:43:45.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.142+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:45.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.146+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338067ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:45.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.146+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:45.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.146+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:45.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.146+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:45.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.146+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:45.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.146+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338063df0 con 0x7f8324008870 2021-03-09T09:43:45.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.146+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338067ee0 secure :-1 s=READY pgs=4365 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344062000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:45.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.146+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401e0e0 con 0x7f8324008870 2021-03-09T09:43:45.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.146+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T09:43:45.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:45.150+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404c8f0 con 0x7f8324008870 2021-03-09T09:43:45.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:43:45 smithi146 bash[13136]: debug 2021-03-09T09:43:45.142+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12361 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:43:50.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:43:50.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:50.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338067ee0 secure :-1 s=READY pgs=4365 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344062000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:50.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338067ee0 secure :-1 s=READY pgs=4365 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344062000).stop 2021-03-09T09:43:50.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338067ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:50.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338067ee0 unknown :-1 s=CLOSED pgs=4365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:50.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:50.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:50.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:50.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006370 con 0x7f8324003400 2021-03-09T09:43:50.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061130 con 0x7f833802d7b0 2021-03-09T09:43:50.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e2c0 con 0x7f8324005e00 2021-03-09T09:43:50.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:43:50.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:50.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:43:50.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:50.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.146+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:50.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.150+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:50.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.150+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:50.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.150+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:50.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.150+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:50.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.150+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:43:50.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.150+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4377 cs=0 l=1 rev1=1 rx=0x7f834400e010 tx=0x7f8344062000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:50.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.150+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404af30 con 0x7f833802d7b0 2021-03-09T09:43:50.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.150+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404af30 con 0x7f833802d7b0 2021-03-09T09:43:50.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:50.154+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404b5f0 con 0x7f833802d7b0 2021-03-09T09:43:50.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:43:50 smithi146 bash[13136]: debug 2021-03-09T09:43:50.146+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12379 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:43:55.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:43:55.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:55.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4377 cs=0 l=1 rev1=1 rx=0x7f834400e010 tx=0x7f8344062000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:55.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4377 cs=0 l=1 rev1=1 rx=0x7f834400e010 tx=0x7f8344062000).stop 2021-03-09T09:43:55.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:55.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:55.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:55.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:55.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:55.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ce80 con 0x7f8324003400 2021-03-09T09:43:55.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324008870 2021-03-09T09:43:55.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061130 con 0x7f8324005e00 2021-03-09T09:43:55.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:43:55.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:55.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:55.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:55.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:55.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:55.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:55.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338018150 con 0x7f8324008870 2021-03-09T09:43:55.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fb0 secure :-1 s=READY pgs=4382 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c065000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:55.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f8324008870 2021-03-09T09:43:55.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.150+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f030 con 0x7f8324008870 2021-03-09T09:43:55.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:43:55.154+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8324008870 2021-03-09T09:43:55.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:43:55 smithi146 bash[13136]: debug 2021-03-09T09:43:55.146+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12412 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:44:00.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338009fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:44:00.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338009fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:00.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fb0 secure :-1 s=READY pgs=4382 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c065000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:00.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fb0 secure :-1 s=READY pgs=4382 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c065000).stop 2021-03-09T09:44:00.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338009fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:00.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fb0 unknown :-1 s=CLOSED pgs=4382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:00.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:00.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:00.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:00.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061130 con 0x7f8324003400 2021-03-09T09:44:00.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:44:00.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e1c0 con 0x7f833802d7b0 2021-03-09T09:44:00.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T09:44:00.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:00.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:44:00.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:00.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.149+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:00.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.153+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:00.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.153+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:00.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.153+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:00.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.153+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:00.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.153+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801d830 con 0x7f833802d7b0 2021-03-09T09:44:00.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.153+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4391 cs=0 l=1 rev1=1 rx=0x7f833c041480 tx=0x7f833c065000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:00.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.153+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c078220 con 0x7f833802d7b0 2021-03-09T09:44:00.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.153+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:44:00.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:00.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00fb10 con 0x7f833802d7b0 2021-03-09T09:44:00.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:44:00 smithi146 bash[13136]: debug 2021-03-09T09:44:00.150+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12431 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:44:05.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.149+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:44:05.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.149+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:05.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.149+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4391 cs=0 l=1 rev1=1 rx=0x7f833c041480 tx=0x7f833c065000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:05.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.149+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4391 cs=0 l=1 rev1=1 rx=0x7f833c041480 tx=0x7f833c065000).stop 2021-03-09T09:44:05.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:05.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.149+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:05.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.149+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:05.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.149+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:05.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.149+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:05.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T09:44:05.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009c30 con 0x7f8324008870 2021-03-09T09:44:05.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061130 con 0x7f8324005e00 2021-03-09T09:44:05.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.153+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:44:05.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.153+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:05.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.153+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:44:05.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.153+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:05.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.153+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:05.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.153+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:05.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.153+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:05.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.153+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:05.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.153+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:05.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.153+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028ba0 con 0x7f8324008870 2021-03-09T09:44:05.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.157+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fb0 secure :-1 s=READY pgs=4400 cs=0 l=1 rev1=1 rx=0x7f83380276a0 tx=0x7f833c065000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:05.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c090220 con 0x7f8324008870 2021-03-09T09:44:05.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8324008870 2021-03-09T09:44:05.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:05.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c041b90 con 0x7f8324008870 2021-03-09T09:44:05.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:44:05 smithi146 bash[13136]: debug 2021-03-09T09:44:05.154+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12459 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:44:10.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.153+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338009fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:44:10.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.153+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338009fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:10.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.153+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fb0 secure :-1 s=READY pgs=4400 cs=0 l=1 rev1=1 rx=0x7f83380276a0 tx=0x7f833c065000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:10.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.153+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fb0 secure :-1 s=READY pgs=4400 cs=0 l=1 rev1=1 rx=0x7f83380276a0 tx=0x7f833c065000).stop 2021-03-09T09:44:10.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.153+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338009fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:10.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.153+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fb0 unknown :-1 s=CLOSED pgs=4400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:10.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.153+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:10.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.153+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:10.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.153+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:10.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.153+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061130 con 0x7f8324003400 2021-03-09T09:44:10.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.153+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380181a0 con 0x7f833802d7b0 2021-03-09T09:44:10.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.153+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T09:44:10.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.157+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:10.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.157+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:44:10.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.157+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:10.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.157+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:44:10.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.157+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:10.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.157+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:10.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.157+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:10.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.157+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:10.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.157+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:10.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.157+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338006010 con 0x7f833802d7b0 2021-03-09T09:44:10.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.157+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4411 cs=0 l=1 rev1=1 rx=0x7f8344072580 tx=0x7f8344062000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:10.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344078e60 con 0x7f833802d7b0 2021-03-09T09:44:10.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833802d7b0 2021-03-09T09:44:10.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:10.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344054d50 con 0x7f833802d7b0 2021-03-09T09:44:10.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:44:10 smithi146 bash[13136]: debug 2021-03-09T09:44:10.154+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12481 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:44:15.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:44:15.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:15.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4411 cs=0 l=1 rev1=1 rx=0x7f8344072580 tx=0x7f8344062000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:15.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4411 cs=0 l=1 rev1=1 rx=0x7f8344072580 tx=0x7f8344062000).stop 2021-03-09T09:44:15.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:15.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:15.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:15.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:15.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:15.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T09:44:15.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024ad0 con 0x7f8324008870 2021-03-09T09:44:15.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806fcc0 con 0x7f8324005e00 2021-03-09T09:44:15.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:44:15.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:15.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:44:15.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:15.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.157+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:15.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.161+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:15.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.161+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:15.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.161+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:15.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.161+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:15.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.161+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338063df0 con 0x7f8324008870 2021-03-09T09:44:15.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.161+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 secure :-1 s=READY pgs=4421 cs=0 l=1 rev1=1 rx=0x7f833c026ff0 tx=0x7f833c05f820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:15.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05f690 con 0x7f8324008870 2021-03-09T09:44:15.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f8324008870 2021-03-09T09:44:15.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:15.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8324008870 2021-03-09T09:44:15.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:44:15 smithi146 bash[13136]: debug 2021-03-09T09:44:15.158+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12510 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:44:20.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:44:20.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:20.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 secure :-1 s=READY pgs=4421 cs=0 l=1 rev1=1 rx=0x7f833c026ff0 tx=0x7f833c05f820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:20.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 secure :-1 s=READY pgs=4421 cs=0 l=1 rev1=1 rx=0x7f833c026ff0 tx=0x7f833c05f820).stop 2021-03-09T09:44:20.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:20.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 unknown :-1 s=CLOSED pgs=4421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:20.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:20.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:20.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:20.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801da80 con 0x7f8324003400 2021-03-09T09:44:20.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f833802d7b0 2021-03-09T09:44:20.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380181a0 con 0x7f8324005e00 2021-03-09T09:44:20.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:44:20.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:20.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:44:20.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:20.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:20.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:20.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:20.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:20.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:20.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.161+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806d580 con 0x7f833802d7b0 2021-03-09T09:44:20.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800a7e0 secure :-1 s=READY pgs=4429 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c05f820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:20.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00f300 con 0x7f833802d7b0 2021-03-09T09:44:20.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00f300 con 0x7f833802d7b0 2021-03-09T09:44:20.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:20.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c060260 con 0x7f833802d7b0 2021-03-09T09:44:20.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:44:20 smithi146 bash[13136]: debug 2021-03-09T09:44:20.162+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12533 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:44:25.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833800a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:44:25.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833800a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:25.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800a7e0 secure :-1 s=READY pgs=4429 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c05f820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:25.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800a7e0 secure :-1 s=READY pgs=4429 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c05f820).stop 2021-03-09T09:44:25.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833800a7e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:25.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800a7e0 unknown :-1 s=CLOSED pgs=4429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:25.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:25.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:25.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:25.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801da80 con 0x7f8324003400 2021-03-09T09:44:25.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014f00 con 0x7f8324008870 2021-03-09T09:44:25.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dfb0 con 0x7f8324005e00 2021-03-09T09:44:25.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:44:25.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:44:25.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:25.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:25.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:25.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:25.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:25.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:25.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:25.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801f050 con 0x7f8324008870 2021-03-09T09:44:25.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 secure :-1 s=READY pgs=4435 cs=0 l=1 rev1=1 rx=0x7f833c063050 tx=0x7f833c05f820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:25.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0614e0 con 0x7f8324008870 2021-03-09T09:44:25.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.173+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c061300 con 0x7f8324008870 2021-03-09T09:44:25.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:25.173+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c029ab0 con 0x7f8324008870 2021-03-09T09:44:25.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:44:25 smithi146 bash[13136]: debug 2021-03-09T09:44:25.162+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12557 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:44:30.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.161+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:44:30.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:30.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 secure :-1 s=READY pgs=4435 cs=0 l=1 rev1=1 rx=0x7f833c063050 tx=0x7f833c05f820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:30.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 secure :-1 s=READY pgs=4435 cs=0 l=1 rev1=1 rx=0x7f833c063050 tx=0x7f833c05f820).stop 2021-03-09T09:44:30.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:30.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 unknown :-1 s=CLOSED pgs=4435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:30.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:30.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:30.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:30.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007ec0 con 0x7f8324003400 2021-03-09T09:44:30.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a520 con 0x7f833802d7b0 2021-03-09T09:44:30.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dfb0 con 0x7f8324005e00 2021-03-09T09:44:30.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:44:30.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:30.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:44:30.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:30.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:30.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:30.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:30.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:30.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:30.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:44:30.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.165+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4442 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344062000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:30.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833802d7b0 2021-03-09T09:44:30.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833802d7b0 2021-03-09T09:44:30.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:30.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344079250 con 0x7f833802d7b0 2021-03-09T09:44:30.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:44:30 smithi146 bash[13136]: debug 2021-03-09T09:44:30.166+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12582 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:44:35.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:44:35.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:35.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4442 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344062000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:35.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4442 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344062000).stop 2021-03-09T09:44:35.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:35.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:35.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:35.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:35.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:35.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014a10 con 0x7f8324003400 2021-03-09T09:44:35.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002d00 con 0x7f8324008870 2021-03-09T09:44:35.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d0e0 con 0x7f8324005e00 2021-03-09T09:44:35.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:44:35.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:35.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:44:35.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:35.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:35.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:35.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:35.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:35.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:35.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338009be0 con 0x7f8324008870 2021-03-09T09:44:35.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007840 secure :-1 s=READY pgs=4452 cs=0 l=1 rev1=1 rx=0x7f833c009980 tx=0x7f833c05f820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:35.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c053f10 con 0x7f8324008870 2021-03-09T09:44:35.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05f690 con 0x7f8324008870 2021-03-09T09:44:35.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:35.173+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055310 con 0x7f8324008870 2021-03-09T09:44:35.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:44:35 smithi146 bash[13136]: debug 2021-03-09T09:44:35.170+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12608 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:44:40.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:44:40.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:40.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007840 secure :-1 s=READY pgs=4452 cs=0 l=1 rev1=1 rx=0x7f833c009980 tx=0x7f833c05f820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:40.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007840 secure :-1 s=READY pgs=4452 cs=0 l=1 rev1=1 rx=0x7f833c009980 tx=0x7f833c05f820).stop 2021-03-09T09:44:40.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:40.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007840 unknown :-1 s=CLOSED pgs=4452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:40.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:40.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:40.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:40.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020b80 con 0x7f8324003400 2021-03-09T09:44:40.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801da80 con 0x7f833802d7b0 2021-03-09T09:44:40.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dfb0 con 0x7f8324005e00 2021-03-09T09:44:40.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:44:40.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:40.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:44:40.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:40.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.169+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:40.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.173+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:40.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.173+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:40.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.173+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:40.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.173+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:40.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.173+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014d00 con 0x7f833802d7b0 2021-03-09T09:44:40.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.173+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4460 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344062000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:40.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.173+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400f7f0 con 0x7f833802d7b0 2021-03-09T09:44:40.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.173+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400f7f0 con 0x7f833802d7b0 2021-03-09T09:44:40.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:40.177+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400f1b0 con 0x7f833802d7b0 2021-03-09T09:44:40.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:44:40 smithi146 bash[13136]: debug 2021-03-09T09:44:40.169+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12644 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:44:45.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.168+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:44:45.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.168+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:45.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.168+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4460 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344062000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:45.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.168+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4460 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344062000).stop 2021-03-09T09:44:45.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:45.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.168+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:45.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.168+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:45.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.168+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:45.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.168+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338013960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:45.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e1c0 con 0x7f8324003400 2021-03-09T09:44:45.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023a20 con 0x7f83380134e0 2021-03-09T09:44:45.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060e60 con 0x7f8324005e00 2021-03-09T09:44:45.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.168+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:44:45.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.168+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:45.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.172+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:44:45.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.172+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:45.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.172+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338013960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:45.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.172+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:45.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.172+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:45.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.172+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:45.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.172+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:45.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.172+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f83380134e0 2021-03-09T09:44:45.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.172+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338013960 secure :-1 s=READY pgs=4468 cs=0 l=1 rev1=1 rx=0x7f83380690a0 tx=0x7f833c05f820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:45.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.172+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0254c0 con 0x7f83380134e0 2021-03-09T09:44:45.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.172+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f83380134e0 2021-03-09T09:44:45.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:45.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f83380134e0 2021-03-09T09:44:45.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:44:45 smithi146 bash[13136]: debug 2021-03-09T09:44:45.173+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12659 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:44:50.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338013960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:44:50.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338013960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:50.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338013960 secure :-1 s=READY pgs=4468 cs=0 l=1 rev1=1 rx=0x7f83380690a0 tx=0x7f833c05f820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:50.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338013960 secure :-1 s=READY pgs=4468 cs=0 l=1 rev1=1 rx=0x7f83380690a0 tx=0x7f833c05f820).stop 2021-03-09T09:44:50.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338013960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:50.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338013960 unknown :-1 s=CLOSED pgs=4468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:50.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:50.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:50.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:50.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dfb0 con 0x7f8324003400 2021-03-09T09:44:50.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d5e0 con 0x7f833802d7b0 2021-03-09T09:44:50.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020130 con 0x7f8324005e00 2021-03-09T09:44:50.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:44:50.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:50.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:44:50.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:50.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:50.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:50.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:50.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:50.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:50.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.172+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338006850 con 0x7f833802d7b0 2021-03-09T09:44:50.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4477 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c05f820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:50.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c06a740 con 0x7f833802d7b0 2021-03-09T09:44:50.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0252e0 con 0x7f833802d7b0 2021-03-09T09:44:50.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06a2d0 con 0x7f833802d7b0 2021-03-09T09:44:50.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:44:50 smithi146 bash[13136]: debug 2021-03-09T09:44:50.173+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12684 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:44:50.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.636+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833801cbc0 msgr2=0x7f8338063f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2021-03-09T09:44:50.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.636+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833801cbc0 msgr2=0x7f8338063f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:50.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.636+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833801cbc0 0x7f8338063f80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f8334004210 tx=0x7f833405c520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:50.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.636+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833801cbc0 0x7f8338063f80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f8334004210 tx=0x7f833405c520).stop 2021-03-09T09:44:50.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.636+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833801cbc0 msgr2=0x7f8338063f80 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T09:44:50.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833801cbc0 msgr2=0x7f8338063f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:50.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.636+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f833801cbc0 0x7f8338063f80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:50.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.636+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f83380134e0 0x7f83380138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:50.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f833c035820 con 0x7f83380134e0 2021-03-09T09:44:50.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.636+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f83380134e0 0x7f83380138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:50.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:50.640+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f83380134e0 0x7f83380138c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f833405e000).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:55.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:44:55.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:55.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4477 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c05f820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:55.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4477 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c05f820).stop 2021-03-09T09:44:55.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:55.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:55.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:55.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338063f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:55.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:55.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:44:55.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380700d0 con 0x7f8324003400 2021-03-09T09:44:55.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801da80 con 0x7f833801cbc0 2021-03-09T09:44:55.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020130 con 0x7f8324005e00 2021-03-09T09:44:55.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338063f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:55.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:55.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:55.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:55.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-03-09T09:44:55.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:44:55.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:55.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:55.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065400 con 0x7f833801cbc0 2021-03-09T09:44:55.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338063f80 secure :-1 s=READY pgs=4486 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344062000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:55.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404e650 con 0x7f833801cbc0 2021-03-09T09:44:55.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833801cbc0 2021-03-09T09:44:55.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:44:55.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400f300 con 0x7f833801cbc0 2021-03-09T09:44:55.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:44:55 smithi146 bash[13136]: debug 2021-03-09T09:44:55.177+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12710 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:45:00.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338063f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:45:00.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338063f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:00.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338063f80 secure :-1 s=READY pgs=4486 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344062000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:00.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338063f80 secure :-1 s=READY pgs=4486 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344062000).stop 2021-03-09T09:45:00.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338063f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:00.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338063f80 unknown :-1 s=CLOSED pgs=4486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:00.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:00.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:00.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:00.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066230 con 0x7f8324003400 2021-03-09T09:45:00.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002f00 con 0x7f833802d7b0 2021-03-09T09:45:00.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dfb0 con 0x7f8324005e00 2021-03-09T09:45:00.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.180+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:45:00.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.180+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:00.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.180+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:45:00.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.180+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:00.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.180+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:00.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.180+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:00.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.180+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:00.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.180+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:00.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.180+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:00.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.180+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ab70 con 0x7f833802d7b0 2021-03-09T09:45:00.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.180+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4494 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344062000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:00.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402ad60 con 0x7f833802d7b0 2021-03-09T09:45:00.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404aa60 con 0x7f833802d7b0 2021-03-09T09:45:00.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:00.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402a2d0 con 0x7f833802d7b0 2021-03-09T09:45:00.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:45:00 smithi146 bash[13136]: debug 2021-03-09T09:45:00.181+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12734 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:45:05.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.180+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:45:05.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.180+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:05.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.180+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4494 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344062000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:05.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.180+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4494 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344062000).stop 2021-03-09T09:45:05.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:05.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.180+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:05.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.180+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338015330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:05.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.180+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:05.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.180+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:05.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020130 con 0x7f8324003400 2021-03-09T09:45:05.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e1c0 con 0x7f833801cbc0 2021-03-09T09:45:05.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021960 con 0x7f8324005e00 2021-03-09T09:45:05.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.180+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:45:05.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.180+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:05.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.180+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:45:05.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.180+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338015330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:05.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.184+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:05.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.184+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:05.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.184+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:05.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.184+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:05.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.184+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:05.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.184+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833801cbc0 2021-03-09T09:45:05.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.184+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338015330 secure :-1 s=READY pgs=4503 cs=0 l=1 rev1=1 rx=0x7f833c018820 tx=0x7f833c05f820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:05.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c06ad80 con 0x7f833801cbc0 2021-03-09T09:45:05.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833801cbc0 2021-03-09T09:45:05.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:05.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833801cbc0 2021-03-09T09:45:05.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:45:05 smithi146 bash[13136]: debug 2021-03-09T09:45:05.181+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12768 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:45:10.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.180+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338015330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:45:10.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.180+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338015330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:10.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.180+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338015330 secure :-1 s=READY pgs=4503 cs=0 l=1 rev1=1 rx=0x7f833c018820 tx=0x7f833c05f820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:10.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.180+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338015330 secure :-1 s=READY pgs=4503 cs=0 l=1 rev1=1 rx=0x7f833c018820 tx=0x7f833c05f820).stop 2021-03-09T09:45:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338015330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338015330 unknown :-1 s=CLOSED pgs=4503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:10.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002f00 con 0x7f8324003400 2021-03-09T09:45:10.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009e30 con 0x7f833802d7b0 2021-03-09T09:45:10.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020130 con 0x7f8324005e00 2021-03-09T09:45:10.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:45:10.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:10.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:45:10.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:10.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:10.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:10.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:10.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:10.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:10.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028ac0 con 0x7f833802d7b0 2021-03-09T09:45:10.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.184+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4512 cs=0 l=1 rev1=1 rx=0x7f8344078670 tx=0x7f8344062000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:10.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404aa60 con 0x7f833802d7b0 2021-03-09T09:45:10.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402be80 con 0x7f833802d7b0 2021-03-09T09:45:10.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:10.192+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ad60 con 0x7f833802d7b0 2021-03-09T09:45:10.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:45:10 smithi146 bash[13136]: debug 2021-03-09T09:45:10.185+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12785 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:45:15.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:45:15.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:15.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4512 cs=0 l=1 rev1=1 rx=0x7f8344078670 tx=0x7f8344062000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:15.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4512 cs=0 l=1 rev1=1 rx=0x7f8344078670 tx=0x7f8344062000).stop 2021-03-09T09:45:15.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:15.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:15.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:15.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:15.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:15.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:45:15.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:15.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009c30 con 0x7f8324003400 2021-03-09T09:45:15.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380274b0 con 0x7f833801cbc0 2021-03-09T09:45:15.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020130 con 0x7f8324005e00 2021-03-09T09:45:15.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:45:15.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:15.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:15.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:15.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:15.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:15.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:15.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801ec80 con 0x7f833801cbc0 2021-03-09T09:45:15.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.188+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833806c690 secure :-1 s=READY pgs=4523 cs=0 l=1 rev1=1 rx=0x7f833c042520 tx=0x7f833c05f820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:15.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.192+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c06ad80 con 0x7f833801cbc0 2021-03-09T09:45:15.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.196+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f833801cbc0 2021-03-09T09:45:15.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:15.196+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054d80 con 0x7f833801cbc0 2021-03-09T09:45:15.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:45:15 smithi146 bash[13136]: debug 2021-03-09T09:45:15.185+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12818 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:45:20.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:45:20.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:20.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833806c690 secure :-1 s=READY pgs=4523 cs=0 l=1 rev1=1 rx=0x7f833c042520 tx=0x7f833c05f820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:20.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833806c690 secure :-1 s=READY pgs=4523 cs=0 l=1 rev1=1 rx=0x7f833c042520 tx=0x7f833c05f820).stop 2021-03-09T09:45:20.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833806c690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:20.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833806c690 unknown :-1 s=CLOSED pgs=4523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:20.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:20.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:20.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:20.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380127a0 con 0x7f8324003400 2021-03-09T09:45:20.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020130 con 0x7f833802d7b0 2021-03-09T09:45:20.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009fd0 con 0x7f8324005e00 2021-03-09T09:45:20.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:45:20.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:20.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:45:20.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:20.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.188+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:20.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.192+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:20.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.192+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:20.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.192+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.192+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.192+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:45:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.192+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4529 cs=0 l=1 rev1=1 rx=0x7f833c022210 tx=0x7f833c05f820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:20.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.192+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00f300 con 0x7f833802d7b0 2021-03-09T09:45:20.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.192+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:45:20.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:20.196+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f833802d7b0 2021-03-09T09:45:20.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:45:20 smithi146 bash[13136]: debug 2021-03-09T09:45:20.189+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12840 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:45:25.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:45:25.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:25.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4529 cs=0 l=1 rev1=1 rx=0x7f833c022210 tx=0x7f833c05f820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:25.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4529 cs=0 l=1 rev1=1 rx=0x7f833c022210 tx=0x7f833c05f820).stop 2021-03-09T09:45:25.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:25.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:25.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:25.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:25.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:25.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020130 con 0x7f8324003400 2021-03-09T09:45:25.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020b80 con 0x7f833801cbc0 2021-03-09T09:45:25.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013170 con 0x7f8324005e00 2021-03-09T09:45:25.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:45:25.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:25.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:45:25.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:25.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:25.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:25.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028ac0 con 0x7f833801cbc0 2021-03-09T09:45:25.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.192+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802db90 secure :-1 s=READY pgs=4539 cs=0 l=1 rev1=1 rx=0x7f833c054d50 tx=0x7f833c05f820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:25.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.196+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018490 con 0x7f833801cbc0 2021-03-09T09:45:25.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.196+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079770 con 0x7f833801cbc0 2021-03-09T09:45:25.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:25.200+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f833801cbc0 2021-03-09T09:45:25.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:45:25 smithi146 bash[13136]: debug 2021-03-09T09:45:25.193+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12864 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:45:30.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833802db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:45:30.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833802db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:30.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802db90 secure :-1 s=READY pgs=4539 cs=0 l=1 rev1=1 rx=0x7f833c054d50 tx=0x7f833c05f820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:30.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802db90 secure :-1 s=READY pgs=4539 cs=0 l=1 rev1=1 rx=0x7f833c054d50 tx=0x7f833c05f820).stop 2021-03-09T09:45:30.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833802db90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:30.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802db90 unknown :-1 s=CLOSED pgs=4539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:30.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:30.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:30.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:30.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009fd0 con 0x7f8324003400 2021-03-09T09:45:30.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020130 con 0x7f833802d7b0 2021-03-09T09:45:30.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015c10 con 0x7f8324005e00 2021-03-09T09:45:30.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:45:30.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:30.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:45:30.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:30.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.195+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:30.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.199+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:30.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:30.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.199+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:30.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:30.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.199+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802d7b0 2021-03-09T09:45:30.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4548 cs=0 l=1 rev1=1 rx=0x7f833c05d010 tx=0x7f833c05f820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:30.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.199+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c078220 con 0x7f833802d7b0 2021-03-09T09:45:30.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.203+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c078220 con 0x7f833802d7b0 2021-03-09T09:45:30.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:30.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02e350 con 0x7f833802d7b0 2021-03-09T09:45:30.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:45:30 smithi146 bash[13136]: debug 2021-03-09T09:45:30.193+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12887 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:45:35.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.195+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:45:35.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.195+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:35.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.195+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4548 cs=0 l=1 rev1=1 rx=0x7f833c05d010 tx=0x7f833c05f820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:35.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.195+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4548 cs=0 l=1 rev1=1 rx=0x7f833c05d010 tx=0x7f833c05f820).stop 2021-03-09T09:45:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026720 con 0x7f8324003400 2021-03-09T09:45:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ab60 con 0x7f833801cbc0 2021-03-09T09:45:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013170 con 0x7f8324005e00 2021-03-09T09:45:35.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:45:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:45:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:35.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:35.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:35.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:35.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:35.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:35.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380728a0 con 0x7f833801cbc0 2021-03-09T09:45:35.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802db90 secure :-1 s=READY pgs=4555 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c05f820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:35.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.203+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c053270 con 0x7f833801cbc0 2021-03-09T09:45:35.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.203+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c054d80 con 0x7f833801cbc0 2021-03-09T09:45:35.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:35.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054d80 con 0x7f833801cbc0 2021-03-09T09:45:35.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:45:35 smithi146 bash[13136]: debug 2021-03-09T09:45:35.197+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12915 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:45:40.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.199+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833802db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:45:40.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.199+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833802db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:40.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802db90 secure :-1 s=READY pgs=4555 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c05f820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:40.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802db90 secure :-1 s=READY pgs=4555 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c05f820).stop 2021-03-09T09:45:40.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.199+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833802db90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:40.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.199+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802db90 unknown :-1 s=CLOSED pgs=4555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:40.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.199+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:40.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.199+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:40.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.199+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:40.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.199+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:45:40.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.199+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020130 con 0x7f8324003400 2021-03-09T09:45:40.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:40.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.199+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380071f0 con 0x7f833802d7b0 2021-03-09T09:45:40.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.199+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a990 con 0x7f8324005e00 2021-03-09T09:45:40.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:40.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.203+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:40.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.203+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:40.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.203+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:40.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.203+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:40.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.203+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028ac0 con 0x7f833802d7b0 2021-03-09T09:45:40.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.203+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4564 cs=0 l=1 rev1=1 rx=0x7f833c0095f0 tx=0x7f833c05f820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:40.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c057d70 con 0x7f833802d7b0 2021-03-09T09:45:40.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c054d80 con 0x7f833802d7b0 2021-03-09T09:45:40.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:40.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01dc50 con 0x7f833802d7b0 2021-03-09T09:45:40.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:45:40 smithi146 bash[13136]: debug 2021-03-09T09:45:40.201+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12938 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:45:45.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:45:45.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:45.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4564 cs=0 l=1 rev1=1 rx=0x7f833c0095f0 tx=0x7f833c05f820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:45.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4564 cs=0 l=1 rev1=1 rx=0x7f833c0095f0 tx=0x7f833c05f820).stop 2021-03-09T09:45:45.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:45.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:45.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:45.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:45.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:45.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a540 con 0x7f8324003400 2021-03-09T09:45:45.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a990 con 0x7f833801cbc0 2021-03-09T09:45:45.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800adc0 con 0x7f8324005e00 2021-03-09T09:45:45.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:45:45.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:45.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:45:45.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:45.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.203+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:45.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.207+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:45.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.207+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:45.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.207+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:45.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.207+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:45.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.207+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020b80 con 0x7f833801cbc0 2021-03-09T09:45:45.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.207+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802e270 secure :-1 s=READY pgs=4573 cs=0 l=1 rev1=1 rx=0x7f8344020fd0 tx=0x7f8344062000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:45.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344054d90 con 0x7f833801cbc0 2021-03-09T09:45:45.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344054ef0 con 0x7f833801cbc0 2021-03-09T09:45:45.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:45.211+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834406b3f0 con 0x7f833801cbc0 2021-03-09T09:45:45.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:45:45 smithi146 bash[13136]: debug 2021-03-09T09:45:45.200+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12965 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:45:50.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833802e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:45:50.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833802e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:50.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802e270 secure :-1 s=READY pgs=4573 cs=0 l=1 rev1=1 rx=0x7f8344020fd0 tx=0x7f8344062000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:50.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802e270 secure :-1 s=READY pgs=4573 cs=0 l=1 rev1=1 rx=0x7f8344020fd0 tx=0x7f8344062000).stop 2021-03-09T09:45:50.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833802e270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:50.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833802e270 unknown :-1 s=CLOSED pgs=4573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:50.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:50.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:50.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:50.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a90 con 0x7f8324003400 2021-03-09T09:45:50.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020130 con 0x7f833802d7b0 2021-03-09T09:45:50.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800adc0 con 0x7f8324005e00 2021-03-09T09:45:50.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:45:50.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:50.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:50.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:45:50.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:50.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:50.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:50.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:50.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:50.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.207+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380728a0 con 0x7f833802d7b0 2021-03-09T09:45:50.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.211+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4581 cs=0 l=1 rev1=1 rx=0x7f833c022210 tx=0x7f833c05f820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:50.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.211+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0094a0 con 0x7f833802d7b0 2021-03-09T09:45:50.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.211+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f030 con 0x7f833802d7b0 2021-03-09T09:45:50.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:50.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833802d7b0 2021-03-09T09:45:50.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:45:50 smithi146 bash[13136]: debug 2021-03-09T09:45:50.204+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 12997 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:45:55.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.207+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:45:55.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.207+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:55.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.207+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4581 cs=0 l=1 rev1=1 rx=0x7f833c022210 tx=0x7f833c05f820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:55.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.207+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4581 cs=0 l=1 rev1=1 rx=0x7f833c022210 tx=0x7f833c05f820).stop 2021-03-09T09:45:55.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:55.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.207+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:55.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.207+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338016440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:55.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.207+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:55.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.207+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:55.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380071f0 con 0x7f8324003400 2021-03-09T09:45:55.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a990 con 0x7f833801cbc0 2021-03-09T09:45:55.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020130 con 0x7f8324005e00 2021-03-09T09:45:55.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.211+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:45:55.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.211+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:55.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.211+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:45:55.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.211+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:55.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.211+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338016440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:55.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.211+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:55.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.211+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:55.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.211+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.211+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.211+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801d840 con 0x7f833801cbc0 2021-03-09T09:45:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.215+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338016440 secure :-1 s=READY pgs=4594 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c022450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c010b80 con 0x7f833801cbc0 2021-03-09T09:45:55.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833801cbc0 2021-03-09T09:45:55.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:45:55.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833801cbc0 2021-03-09T09:45:55.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:45:55 smithi146 bash[13136]: debug 2021-03-09T09:45:55.208+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13026 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:46:00.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.211+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338016440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:46:00.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.211+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338016440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:00.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.211+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338016440 secure :-1 s=READY pgs=4594 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c022450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:00.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.211+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338016440 secure :-1 s=READY pgs=4594 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c022450).stop 2021-03-09T09:46:00.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.211+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338016440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:00.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.211+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338016440 unknown :-1 s=CLOSED pgs=4594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:00.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.211+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:00.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.211+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:00.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.211+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:00.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.211+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d5c0 con 0x7f8324003400 2021-03-09T09:46:00.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.211+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013170 con 0x7f833802d7b0 2021-03-09T09:46:00.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.211+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020130 con 0x7f8324005e00 2021-03-09T09:46:00.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.211+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:46:00.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.211+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:00.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.211+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:46:00.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.211+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:00.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.215+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:00.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.215+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:00.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.215+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:00.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.215+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:00.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.215+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:00.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.215+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380728a0 con 0x7f833802d7b0 2021-03-09T09:46:00.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.215+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4601 cs=0 l=1 rev1=1 rx=0x7f834404c5a0 tx=0x7f8344062000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:00.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833802d7b0 2021-03-09T09:46:00.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404a800 con 0x7f833802d7b0 2021-03-09T09:46:00.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:00.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402adb0 con 0x7f833802d7b0 2021-03-09T09:46:00.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:46:00 smithi146 bash[13136]: debug 2021-03-09T09:46:00.212+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13044 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:46:05.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:46:05.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:05.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4601 cs=0 l=1 rev1=1 rx=0x7f834404c5a0 tx=0x7f8344062000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:05.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4601 cs=0 l=1 rev1=1 rx=0x7f834404c5a0 tx=0x7f8344062000).stop 2021-03-09T09:46:05.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:05.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:05.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:05.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:05.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:05.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015c10 con 0x7f8324003400 2021-03-09T09:46:05.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f833801cbc0 2021-03-09T09:46:05.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:46:05.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:46:05.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:05.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:46:05.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:05.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:05.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:05.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:05.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:05.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:05.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026af0 con 0x7f833801cbc0 2021-03-09T09:46:05.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.215+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005810 secure :-1 s=READY pgs=4607 cs=0 l=1 rev1=1 rx=0x7f833c078670 tx=0x7f833c010e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:05.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01d8e0 con 0x7f833801cbc0 2021-03-09T09:46:05.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01d3a0 con 0x7f833801cbc0 2021-03-09T09:46:05.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:05.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01d570 con 0x7f833801cbc0 2021-03-09T09:46:05.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:46:05 smithi146 bash[13136]: debug 2021-03-09T09:46:05.212+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13071 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:46:10.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.215+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338005810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:46:10.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.215+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338005810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:10.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.215+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005810 secure :-1 s=READY pgs=4607 cs=0 l=1 rev1=1 rx=0x7f833c078670 tx=0x7f833c010e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:10.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.215+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005810 secure :-1 s=READY pgs=4607 cs=0 l=1 rev1=1 rx=0x7f833c078670 tx=0x7f833c010e50).stop 2021-03-09T09:46:10.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338005810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:10.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.215+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005810 unknown :-1 s=CLOSED pgs=4607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:10.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.215+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:10.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.215+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:10.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.215+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:10.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a990 con 0x7f8324003400 2021-03-09T09:46:10.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014da0 con 0x7f833802d7b0 2021-03-09T09:46:10.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014690 con 0x7f8324005e00 2021-03-09T09:46:10.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.215+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:46:10.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.215+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:10.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.215+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:46:10.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.215+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:10.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:10.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.219+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:10.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:10.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.219+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:10.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:10.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.219+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380728a0 con 0x7f833802d7b0 2021-03-09T09:46:10.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4616 cs=0 l=1 rev1=1 rx=0x7f833c041b60 tx=0x7f833c010e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:10.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c057760 con 0x7f833802d7b0 2021-03-09T09:46:10.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c4d0 con 0x7f833802d7b0 2021-03-09T09:46:10.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:10.227+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c078300 con 0x7f833802d7b0 2021-03-09T09:46:10.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:46:10 smithi146 bash[13136]: debug 2021-03-09T09:46:10.216+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13091 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:46:15.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.218+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:46:15.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.218+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:15.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.218+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4616 cs=0 l=1 rev1=1 rx=0x7f833c041b60 tx=0x7f833c010e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:15.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.218+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4616 cs=0 l=1 rev1=1 rx=0x7f833c041b60 tx=0x7f833c010e50).stop 2021-03-09T09:46:15.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.218+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:15.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.218+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:15.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.218+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:15.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.218+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:15.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.218+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:15.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.218+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066e70 con 0x7f8324003400 2021-03-09T09:46:15.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.218+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f833801cbc0 2021-03-09T09:46:15.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.218+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013170 con 0x7f8324005e00 2021-03-09T09:46:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.218+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:46:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.218+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:46:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.218+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.218+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:15.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.222+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:15.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.222+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:15.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.222+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:15.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.222+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:15.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.222+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:15.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.222+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806d510 con 0x7f833801cbc0 2021-03-09T09:46:15.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.222+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005810 secure :-1 s=READY pgs=4625 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c010e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:15.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.222+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c020720 con 0x7f833801cbc0 2021-03-09T09:46:15.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.222+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833801cbc0 2021-03-09T09:46:15.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:15.226+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c026380 con 0x7f833801cbc0 2021-03-09T09:46:15.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:46:15 smithi146 bash[13136]: debug 2021-03-09T09:46:15.220+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13126 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:46:20.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338005810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:46:20.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338005810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:20.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005810 secure :-1 s=READY pgs=4625 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c010e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:20.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005810 secure :-1 s=READY pgs=4625 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c010e50).stop 2021-03-09T09:46:20.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338005810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:20.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005810 unknown :-1 s=CLOSED pgs=4625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:20.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:20.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:20.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:20.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061110 con 0x7f8324003400 2021-03-09T09:46:20.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f833802d7b0 2021-03-09T09:46:20.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380127d0 con 0x7f8324005e00 2021-03-09T09:46:20.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:46:20.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:46:20.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:20.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:20.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:20.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.222+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380728a0 con 0x7f833802d7b0 2021-03-09T09:46:20.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.226+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4633 cs=0 l=1 rev1=1 rx=0x7f833c0187d0 tx=0x7f833c010e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:20.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.226+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05c380 con 0x7f833802d7b0 2021-03-09T09:46:20.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.226+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05c380 con 0x7f833802d7b0 2021-03-09T09:46:20.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:20.230+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833802d7b0 2021-03-09T09:46:20.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:46:20 smithi146 bash[13136]: debug 2021-03-09T09:46:20.220+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13150 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:46:25.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.222+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:46:25.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.222+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:25.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.222+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4633 cs=0 l=1 rev1=1 rx=0x7f833c0187d0 tx=0x7f833c010e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:25.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.222+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4633 cs=0 l=1 rev1=1 rx=0x7f833c0187d0 tx=0x7f833c010e50).stop 2021-03-09T09:46:25.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.222+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:25.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.222+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:25.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.222+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:25.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.226+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:25.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.226+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:25.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.226+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007430 con 0x7f8324003400 2021-03-09T09:46:25.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.226+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f833801cbc0 2021-03-09T09:46:25.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.226+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a130 con 0x7f8324005e00 2021-03-09T09:46:25.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.226+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:46:25.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.226+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:46:25.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.226+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:25.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.226+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:25.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.226+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:25.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.226+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:25.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.226+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:25.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.226+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:25.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.226+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:25.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.226+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801f8d0 con 0x7f833801cbc0 2021-03-09T09:46:25.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.226+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005ec0 secure :-1 s=READY pgs=4642 cs=0 l=1 rev1=1 rx=0x7f833c00f9e0 tx=0x7f833c010e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:25.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.230+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01d3a0 con 0x7f833801cbc0 2021-03-09T09:46:25.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.230+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833801cbc0 2021-03-09T09:46:25.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:25.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05ca60 con 0x7f833801cbc0 2021-03-09T09:46:25.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:46:25 smithi146 bash[13136]: debug 2021-03-09T09:46:25.224+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13179 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:46:30.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338005ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:46:30.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338005ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:30.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005ec0 secure :-1 s=READY pgs=4642 cs=0 l=1 rev1=1 rx=0x7f833c00f9e0 tx=0x7f833c010e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:30.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005ec0 secure :-1 s=READY pgs=4642 cs=0 l=1 rev1=1 rx=0x7f833c00f9e0 tx=0x7f833c010e50).stop 2021-03-09T09:46:30.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338005ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:30.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338005ec0 unknown :-1 s=CLOSED pgs=4642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:30.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:30.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:30.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:30.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e020 con 0x7f8324003400 2021-03-09T09:46:30.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a90 con 0x7f833802d7b0 2021-03-09T09:46:30.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061160 con 0x7f8324005e00 2021-03-09T09:46:30.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:46:30.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:30.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:46:30.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:30.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.226+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:30.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.230+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:30.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.230+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:30.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.230+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:30.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.230+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:30.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.230+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028fe0 con 0x7f833802d7b0 2021-03-09T09:46:30.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.230+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4652 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f8344062000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:30.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833802d7b0 2021-03-09T09:46:30.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.238+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833802d7b0 2021-03-09T09:46:30.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:30.238+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404b680 con 0x7f833802d7b0 2021-03-09T09:46:30.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:46:30 smithi146 bash[13136]: debug 2021-03-09T09:46:30.228+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13194 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:46:35.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:46:35.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:35.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4652 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f8344062000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:35.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4652 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f8344062000).stop 2021-03-09T09:46:35.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:35.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:35.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:35.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833800fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:35.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:35.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007430 con 0x7f8324003400 2021-03-09T09:46:35.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f833801cbc0 2021-03-09T09:46:35.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338029020 con 0x7f8324005e00 2021-03-09T09:46:35.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:46:35.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:35.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:46:35.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:35.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833800fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:35.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:35.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:35.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:35.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:35.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.230+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380728a0 con 0x7f833801cbc0 2021-03-09T09:46:35.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.234+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833800fa70 secure :-1 s=READY pgs=4659 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:35.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344032140 con 0x7f833801cbc0 2021-03-09T09:46:35.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344032e90 con 0x7f833801cbc0 2021-03-09T09:46:35.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:35.238+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ade0 con 0x7f833801cbc0 2021-03-09T09:46:35.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:46:35 smithi146 bash[13136]: debug 2021-03-09T09:46:35.232+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13224 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:46:40.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833800fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:46:40.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833800fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:40.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833800fa70 secure :-1 s=READY pgs=4659 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:40.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833800fa70 secure :-1 s=READY pgs=4659 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344004c60).stop 2021-03-09T09:46:40.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833800fa70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:40.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833800fa70 unknown :-1 s=CLOSED pgs=4659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:40.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:40.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:40.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:40.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007430 con 0x7f8324003400 2021-03-09T09:46:40.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014690 con 0x7f833802d7b0 2021-03-09T09:46:40.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8324005e00 2021-03-09T09:46:40.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:46:40.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:40.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:40.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:46:40.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:40.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:40.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:40.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:40.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:40.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338027cf0 con 0x7f833802d7b0 2021-03-09T09:46:40.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4667 cs=0 l=1 rev1=1 rx=0x7f833c013b00 tx=0x7f833c010e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:40.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c053270 con 0x7f833802d7b0 2021-03-09T09:46:40.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833802d7b0 2021-03-09T09:46:40.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:40.238+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c078220 con 0x7f833802d7b0 2021-03-09T09:46:40.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:46:40 smithi146 bash[13136]: debug 2021-03-09T09:46:40.236+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13256 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:46:45.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.234+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:46:45.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.234+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:45.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.234+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4667 cs=0 l=1 rev1=1 rx=0x7f833c013b00 tx=0x7f833c010e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:45.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.234+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4667 cs=0 l=1 rev1=1 rx=0x7f833c013b00 tx=0x7f833c010e50).stop 2021-03-09T09:46:45.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:45.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.234+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:45.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.234+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:45.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.234+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:45.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.234+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:45.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bb80 con 0x7f8324003400 2021-03-09T09:46:45.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007430 con 0x7f833801cbc0 2021-03-09T09:46:45.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fe10 con 0x7f8324005e00 2021-03-09T09:46:45.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.234+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:46:45.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.234+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:45.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.238+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:46:45.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.238+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:45.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.238+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:45.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.238+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:45.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.238+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:45.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.238+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:45.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.238+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:45.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.238+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380728a0 con 0x7f833801cbc0 2021-03-09T09:46:45.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.238+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 secure :-1 s=READY pgs=4678 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c010e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:45.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.242+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c026590 con 0x7f833801cbc0 2021-03-09T09:46:45.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.242+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c036340 con 0x7f833801cbc0 2021-03-09T09:46:45.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:45.246+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c026380 con 0x7f833801cbc0 2021-03-09T09:46:45.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:46:45 smithi146 bash[13136]: debug 2021-03-09T09:46:45.240+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13278 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:46:50.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:46:50.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:50.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 secure :-1 s=READY pgs=4678 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c010e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:50.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 secure :-1 s=READY pgs=4678 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c010e50).stop 2021-03-09T09:46:50.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:50.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 unknown :-1 s=CLOSED pgs=4678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:50.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:50.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:50.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:50.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:46:50.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bb80 con 0x7f8324003400 2021-03-09T09:46:50.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:50.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:50.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:46:50.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fe70 con 0x7f833802d7b0 2021-03-09T09:46:50.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:46:50.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.238+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:50.246 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.242+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:50.246 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.242+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:50.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.242+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:50.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.242+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:50.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.242+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338002b10 con 0x7f833802d7b0 2021-03-09T09:46:50.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.242+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4688 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c04d460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:50.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.246+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01c4d0 con 0x7f833802d7b0 2021-03-09T09:46:50.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.246+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c4d0 con 0x7f833802d7b0 2021-03-09T09:46:50.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:50.246+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0792c0 con 0x7f833802d7b0 2021-03-09T09:46:50.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:46:50 smithi146 bash[13136]: debug 2021-03-09T09:46:50.239+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13307 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:46:55.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.238+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:46:55.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.238+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:55.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.238+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4688 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c04d460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:55.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.238+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4688 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c04d460).stop 2021-03-09T09:46:55.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.238+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:55.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.238+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:55.246 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.238+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:55.246 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.242+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:55.246 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.242+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:55.246 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.242+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013170 con 0x7f8324003400 2021-03-09T09:46:55.246 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.242+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f833801cbc0 2021-03-09T09:46:55.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.242+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007430 con 0x7f8324005e00 2021-03-09T09:46:55.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.242+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:46:55.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.242+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:55.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.242+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:55.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.242+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:46:55.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.242+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:55.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.242+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:55.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.242+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:55.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.242+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:55.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.242+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:55.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.242+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806dfd0 con 0x7f833801cbc0 2021-03-09T09:46:55.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.242+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 secure :-1 s=READY pgs=4694 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c04d460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:55.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.246+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c053270 con 0x7f833801cbc0 2021-03-09T09:46:55.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.246+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833801cbc0 2021-03-09T09:46:55.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:46:55.250+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c062350 con 0x7f833801cbc0 2021-03-09T09:46:55.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:46:55 smithi146 bash[13136]: debug 2021-03-09T09:46:55.243+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13320 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:47:00.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.241+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:47:00.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.241+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:00.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.241+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 secure :-1 s=READY pgs=4694 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c04d460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:00.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.241+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 secure :-1 s=READY pgs=4694 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c04d460).stop 2021-03-09T09:47:00.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.241+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:00.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.241+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 unknown :-1 s=CLOSED pgs=4694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:00.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.241+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:00.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.241+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:00.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.241+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:00.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.241+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007430 con 0x7f8324003400 2021-03-09T09:47:00.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.241+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e220 con 0x7f833802d7b0 2021-03-09T09:47:00.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.241+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f4f0 con 0x7f8324005e00 2021-03-09T09:47:00.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.241+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:47:00.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.241+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:00.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.241+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:00.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.245+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:47:00.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.245+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:00.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.245+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:00.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.245+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:00.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.245+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:00.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.245+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:00.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.245+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801f8d0 con 0x7f833802d7b0 2021-03-09T09:47:00.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.245+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4702 cs=0 l=1 rev1=1 rx=0x7f833c009980 tx=0x7f833c04d460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:00.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.249+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c029800 con 0x7f833802d7b0 2021-03-09T09:47:00.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.249+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0538b0 con 0x7f833802d7b0 2021-03-09T09:47:00.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:00.253+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c003a70 con 0x7f833802d7b0 2021-03-09T09:47:00.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:47:00 smithi146 bash[13136]: debug 2021-03-09T09:47:00.247+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13354 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:47:05.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:47:05.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:05.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4702 cs=0 l=1 rev1=1 rx=0x7f833c009980 tx=0x7f833c04d460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:05.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4702 cs=0 l=1 rev1=1 rx=0x7f833c009980 tx=0x7f833c04d460).stop 2021-03-09T09:47:05.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:05.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:05.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:05.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:05.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:05.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002f00 con 0x7f8324003400 2021-03-09T09:47:05.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f833801cbc0 2021-03-09T09:47:05.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007430 con 0x7f8324005e00 2021-03-09T09:47:05.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:47:05.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:05.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:47:05.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:05.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.245+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:05.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.249+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:05.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.249+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:05.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.249+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:05.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.249+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:05.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.249+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007230 con 0x7f833801cbc0 2021-03-09T09:47:05.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.249+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 secure :-1 s=READY pgs=4715 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c04d460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:05.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.249+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c060260 con 0x7f833801cbc0 2021-03-09T09:47:05.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.249+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833801cbc0 2021-03-09T09:47:05.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:05.253+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f833801cbc0 2021-03-09T09:47:05.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:47:05 smithi146 bash[13136]: debug 2021-03-09T09:47:05.247+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13372 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:47:10.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.245+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:47:10.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.245+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:10.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.245+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 secure :-1 s=READY pgs=4715 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c04d460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:10.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.245+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 secure :-1 s=READY pgs=4715 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c04d460).stop 2021-03-09T09:47:10.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.245+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:10.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.245+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007800 unknown :-1 s=CLOSED pgs=4715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:10.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.245+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:10.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.249+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.249+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.249+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026720 con 0x7f8324003400 2021-03-09T09:47:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.249+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fe70 con 0x7f833802d7b0 2021-03-09T09:47:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.249+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:47:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.249+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:47:10.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.249+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:10.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.249+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:47:10.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.249+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:10.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.249+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:10.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.249+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:10.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.249+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:10.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.249+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:10.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.249+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:10.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.249+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010d80 con 0x7f833802d7b0 2021-03-09T09:47:10.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.249+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4720 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c04d460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:10.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.253+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0414f0 con 0x7f833802d7b0 2021-03-09T09:47:10.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.253+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c057620 con 0x7f833802d7b0 2021-03-09T09:47:10.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:10.253+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c026380 con 0x7f833802d7b0 2021-03-09T09:47:10.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:47:10 smithi146 bash[13136]: debug 2021-03-09T09:47:10.251+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13404 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:47:15.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:47:15.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:15.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4720 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c04d460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:15.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4720 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c04d460).stop 2021-03-09T09:47:15.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:15.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:15.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:15.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:15.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:15.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013170 con 0x7f8324003400 2021-03-09T09:47:15.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ad20 con 0x7f833801cbc0 2021-03-09T09:47:15.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e220 con 0x7f8324005e00 2021-03-09T09:47:15.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:47:15.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:47:15.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:15.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:15.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.253+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:15.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.257+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:15.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:15.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.257+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:15.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:15.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.257+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338023a20 con 0x7f833801cbc0 2021-03-09T09:47:15.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 secure :-1 s=READY pgs=4732 cs=0 l=1 rev1=1 rx=0x7f833c060a80 tx=0x7f833c04d460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:15.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c053650 con 0x7f833801cbc0 2021-03-09T09:47:15.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c053450 con 0x7f833801cbc0 2021-03-09T09:47:15.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:15.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c060260 con 0x7f833801cbc0 2021-03-09T09:47:15.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:47:15 smithi146 bash[13136]: debug 2021-03-09T09:47:15.255+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13430 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:47:20.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:47:20.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:20.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 secure :-1 s=READY pgs=4732 cs=0 l=1 rev1=1 rx=0x7f833c060a80 tx=0x7f833c04d460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:20.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 secure :-1 s=READY pgs=4732 cs=0 l=1 rev1=1 rx=0x7f833c060a80 tx=0x7f833c04d460).stop 2021-03-09T09:47:20.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:20.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 unknown :-1 s=CLOSED pgs=4732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:20.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:20.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:20.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:20.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007430 con 0x7f8324003400 2021-03-09T09:47:20.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c9a0 con 0x7f833802d7b0 2021-03-09T09:47:20.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:47:20.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:47:20.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:20.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:47:20.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:20.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.253+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:20.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.257+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:20.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:20.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.257+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:20.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:20.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.257+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b7d0 con 0x7f833802d7b0 2021-03-09T09:47:20.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4737 cs=0 l=1 rev1=1 rx=0x7f833c01ab10 tx=0x7f833c04d460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:20.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c003c80 con 0x7f833802d7b0 2021-03-09T09:47:20.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05fbd0 con 0x7f833802d7b0 2021-03-09T09:47:20.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:20.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c060260 con 0x7f833802d7b0 2021-03-09T09:47:20.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:47:20 smithi146 bash[13136]: debug 2021-03-09T09:47:20.255+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13456 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:47:25.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.253+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:47:25.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.253+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:25.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.253+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4737 cs=0 l=1 rev1=1 rx=0x7f833c01ab10 tx=0x7f833c04d460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:25.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.253+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4737 cs=0 l=1 rev1=1 rx=0x7f833c01ab10 tx=0x7f833c04d460).stop 2021-03-09T09:47:25.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:25.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:25.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380152a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:25.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:25.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:25.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f4b0 con 0x7f8324003400 2021-03-09T09:47:25.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002f00 con 0x7f833801cbc0 2021-03-09T09:47:25.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324005e00 2021-03-09T09:47:25.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:47:25.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:25.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:47:25.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:25.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:25.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:25.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:25.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:25.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:25.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a0a0 con 0x7f833801cbc0 2021-03-09T09:47:25.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380152a0 secure :-1 s=READY pgs=4746 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c04d460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:25.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c026e10 con 0x7f833801cbc0 2021-03-09T09:47:25.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03f070 con 0x7f833801cbc0 2021-03-09T09:47:25.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:25.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c060ab0 con 0x7f833801cbc0 2021-03-09T09:47:25.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:47:25 smithi146 bash[13136]: debug 2021-03-09T09:47:25.259+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13483 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:47:30.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.257+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f83380152a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:47:30.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.257+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f83380152a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:30.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380152a0 secure :-1 s=READY pgs=4746 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c04d460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:30.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380152a0 secure :-1 s=READY pgs=4746 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c04d460).stop 2021-03-09T09:47:30.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f83380152a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:30.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.257+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380152a0 unknown :-1 s=CLOSED pgs=4746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:30.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.257+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:30.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.257+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:30.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.257+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:30.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007430 con 0x7f8324003400 2021-03-09T09:47:30.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002f00 con 0x7f833802d7b0 2021-03-09T09:47:30.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068990 con 0x7f8324005e00 2021-03-09T09:47:30.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.257+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:47:30.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:30.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.257+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:47:30.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.257+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:30.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.261+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:30.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.261+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:30.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.261+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:30.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.261+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:30.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.261+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:30.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.261+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024c50 con 0x7f833802d7b0 2021-03-09T09:47:30.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.261+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4754 cs=0 l=1 rev1=1 rx=0x7f833c04d430 tx=0x7f833c037000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:30.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c060470 con 0x7f833802d7b0 2021-03-09T09:47:30.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c023a60 con 0x7f833802d7b0 2021-03-09T09:47:30.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:30.265+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c060260 con 0x7f833802d7b0 2021-03-09T09:47:30.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:47:30 smithi146 bash[13136]: debug 2021-03-09T09:47:30.263+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13509 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:47:35.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:47:35.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:35.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4754 cs=0 l=1 rev1=1 rx=0x7f833c04d430 tx=0x7f833c037000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:35.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4754 cs=0 l=1 rev1=1 rx=0x7f833c04d430 tx=0x7f833c037000).stop 2021-03-09T09:47:35.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:35.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:35.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380152a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:35.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:35.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:35.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324003400 2021-03-09T09:47:35.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e0c0 con 0x7f833801cbc0 2021-03-09T09:47:35.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068990 con 0x7f8324005e00 2021-03-09T09:47:35.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:47:35.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:35.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:47:35.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:35.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:35.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:35.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:35.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:35.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:35.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.261+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068e80 con 0x7f833801cbc0 2021-03-09T09:47:35.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.265+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380152a0 secure :-1 s=READY pgs=4763 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c037000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:35.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.265+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c060470 con 0x7f833801cbc0 2021-03-09T09:47:35.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.265+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00fda0 con 0x7f833801cbc0 2021-03-09T09:47:35.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:35.269+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c003a50 con 0x7f833801cbc0 2021-03-09T09:47:35.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:47:35 smithi146 bash[13136]: debug 2021-03-09T09:47:35.263+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13528 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:47:40.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.261+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f83380152a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:47:40.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.261+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f83380152a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:40.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.261+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380152a0 secure :-1 s=READY pgs=4763 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c037000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:40.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.261+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380152a0 secure :-1 s=READY pgs=4763 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c037000).stop 2021-03-09T09:47:40.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f83380152a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:40.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.261+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380152a0 unknown :-1 s=CLOSED pgs=4763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:40.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.261+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:40.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.261+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:40.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.261+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:40.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.269+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380170c0 con 0x7f8324003400 2021-03-09T09:47:40.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.269+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806f030 con 0x7f833802d7b0 2021-03-09T09:47:40.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.269+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e540 con 0x7f8324005e00 2021-03-09T09:47:40.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.269+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:47:40.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.269+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:40.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.269+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:40.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.269+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:47:40.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.269+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:40.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.273+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:40.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.273+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:40.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.273+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:40.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.273+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:40.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.273+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800e310 con 0x7f833802d7b0 2021-03-09T09:47:40.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.273+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4772 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f8344004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:40.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.273+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833802d7b0 2021-03-09T09:47:40.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.273+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833802d7b0 2021-03-09T09:47:40.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:40.273+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404c8f0 con 0x7f833802d7b0 2021-03-09T09:47:40.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:47:40 smithi146 bash[13136]: debug 2021-03-09T09:47:40.267+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13562 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:47:45.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:47:45.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:45.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4772 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f8344004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:45.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4772 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f8344004c60).stop 2021-03-09T09:47:45.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:45.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:45.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:45.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:45.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:45.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e0c0 con 0x7f8324003400 2021-03-09T09:47:45.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f833801cbc0 2021-03-09T09:47:45.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020540 con 0x7f8324005e00 2021-03-09T09:47:45.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:47:45.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:47:45.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:45.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:45.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:45.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:45.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:45.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:45.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:45.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b7d0 con 0x7f833801cbc0 2021-03-09T09:47:45.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.268+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 secure :-1 s=READY pgs=4783 cs=0 l=1 rev1=1 rx=0x7f833c057ac0 tx=0x7f833c037000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:45.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.272+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0412e0 con 0x7f833801cbc0 2021-03-09T09:47:45.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.272+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c034200 con 0x7f833801cbc0 2021-03-09T09:47:45.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:45.276+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f833801cbc0 2021-03-09T09:47:45.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:47:45 smithi146 bash[13136]: debug 2021-03-09T09:47:45.267+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13575 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:47:50.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:47:50.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:50.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 secure :-1 s=READY pgs=4783 cs=0 l=1 rev1=1 rx=0x7f833c057ac0 tx=0x7f833c037000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:50.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 secure :-1 s=READY pgs=4783 cs=0 l=1 rev1=1 rx=0x7f833c057ac0 tx=0x7f833c037000).stop 2021-03-09T09:47:50.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:50.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 unknown :-1 s=CLOSED pgs=4783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:50.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:50.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:50.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:50.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e0c0 con 0x7f8324003400 2021-03-09T09:47:50.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380728e0 con 0x7f833802d7b0 2021-03-09T09:47:50.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a130 con 0x7f8324005e00 2021-03-09T09:47:50.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:47:50.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:50.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:47:50.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:50.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:50.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:50.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:50.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:50.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:50.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.268+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800bd40 con 0x7f833802d7b0 2021-03-09T09:47:50.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.272+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4789 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c037000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:50.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.272+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00f300 con 0x7f833802d7b0 2021-03-09T09:47:50.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.272+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01d760 con 0x7f833802d7b0 2021-03-09T09:47:50.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:50.276+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c065070 con 0x7f833802d7b0 2021-03-09T09:47:50.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:47:50 smithi146 bash[13136]: debug 2021-03-09T09:47:50.271+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13602 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:47:55.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.268+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:47:55.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.268+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:55.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.268+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4789 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c037000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:55.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.268+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4789 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c037000).stop 2021-03-09T09:47:55.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.268+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:55.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.268+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:55.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.268+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:55.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.268+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:55.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.268+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:55.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.268+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e0c0 con 0x7f8324003400 2021-03-09T09:47:55.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.268+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021e50 con 0x7f833801cbc0 2021-03-09T09:47:55.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.268+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8324005e00 2021-03-09T09:47:55.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.268+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:47:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.268+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.268+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:47:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.268+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.272+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.272+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.272+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.272+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.272+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.272+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ed40 con 0x7f833801cbc0 2021-03-09T09:47:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.272+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 secure :-1 s=READY pgs=4798 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.276+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400fdf0 con 0x7f833801cbc0 2021-03-09T09:47:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.276+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833801cbc0 2021-03-09T09:47:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:47:55.280+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f833801cbc0 2021-03-09T09:47:55.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:47:55 smithi146 bash[13136]: debug 2021-03-09T09:47:55.275+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13630 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:48:00.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:48:00.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:00.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 secure :-1 s=READY pgs=4798 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:00.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 secure :-1 s=READY pgs=4798 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344004c60).stop 2021-03-09T09:48:00.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:00.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007720 unknown :-1 s=CLOSED pgs=4798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:00.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:00.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:00.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:00.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e0c0 con 0x7f8324003400 2021-03-09T09:48:00.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022220 con 0x7f833802d7b0 2021-03-09T09:48:00.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019150 con 0x7f8324005e00 2021-03-09T09:48:00.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:48:00.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:00.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:48:00.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:00.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.272+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:00.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.276+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:00.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.276+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:00.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.276+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:00.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.276+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:00.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.276+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f833802d7b0 2021-03-09T09:48:00.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.276+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4807 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f8344004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:00.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.276+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440793f0 con 0x7f833802d7b0 2021-03-09T09:48:00.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.276+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f833802d7b0 2021-03-09T09:48:00.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:00.280+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834406b760 con 0x7f833802d7b0 2021-03-09T09:48:00.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:48:00 smithi146 bash[13136]: debug 2021-03-09T09:48:00.274+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13653 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:48:05.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.272+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:48:05.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.272+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:05.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.272+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4807 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f8344004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:05.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.272+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4807 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f8344004c60).stop 2021-03-09T09:48:05.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.272+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:05.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.272+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:05.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.276+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:05.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.276+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338013db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:05.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.276+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:05.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.276+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062380 con 0x7f8324003400 2021-03-09T09:48:05.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.276+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f833801cbc0 2021-03-09T09:48:05.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.276+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016010 con 0x7f8324005e00 2021-03-09T09:48:05.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.276+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:48:05.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.276+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:05.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.276+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:48:05.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.276+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:05.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.276+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338013db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:05.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.276+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:05.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.276+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:05.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.276+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:05.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.276+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:05.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.276+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022220 con 0x7f833801cbc0 2021-03-09T09:48:05.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.280+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338013db0 secure :-1 s=READY pgs=4815 cs=0 l=1 rev1=1 rx=0x7f83380428c0 tx=0x7f8344004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:05.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.280+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401f120 con 0x7f833801cbc0 2021-03-09T09:48:05.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.280+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f833801cbc0 2021-03-09T09:48:05.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:05.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344078bd0 con 0x7f833801cbc0 2021-03-09T09:48:05.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:48:05 smithi146 bash[13136]: debug 2021-03-09T09:48:05.278+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13678 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:48:10.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338013db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:48:10.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338013db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338013db0 secure :-1 s=READY pgs=4815 cs=0 l=1 rev1=1 rx=0x7f83380428c0 tx=0x7f8344004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338013db0 secure :-1 s=READY pgs=4815 cs=0 l=1 rev1=1 rx=0x7f83380428c0 tx=0x7f8344004c60).stop 2021-03-09T09:48:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338013db0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338013db0 unknown :-1 s=CLOSED pgs=4815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a5a0 con 0x7f8324003400 2021-03-09T09:48:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002f00 con 0x7f833802d7b0 2021-03-09T09:48:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e0c0 con 0x7f8324005e00 2021-03-09T09:48:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:48:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:48:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.276+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.280+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.280+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.280+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:10.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.280+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:10.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.280+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806ce00 con 0x7f833802d7b0 2021-03-09T09:48:10.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.280+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4824 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:10.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344066e80 con 0x7f833802d7b0 2021-03-09T09:48:10.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344066e80 con 0x7f833802d7b0 2021-03-09T09:48:10.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:10.288+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400f1b0 con 0x7f833802d7b0 2021-03-09T09:48:10.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:48:10 smithi146 bash[13136]: debug 2021-03-09T09:48:10.282+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13703 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:48:15.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:48:15.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:15.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4824 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:15.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4824 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344049b10).stop 2021-03-09T09:48:15.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:15.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:15.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:15.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:15.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:15.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019150 con 0x7f8324003400 2021-03-09T09:48:15.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b7d0 con 0x7f833801cbc0 2021-03-09T09:48:15.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a990 con 0x7f8324005e00 2021-03-09T09:48:15.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:48:15.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:15.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:48:15.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:15.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:15.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:15.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:15.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:15.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:15.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801d6c0 con 0x7f833801cbc0 2021-03-09T09:48:15.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380100b0 secure :-1 s=READY pgs=4832 cs=0 l=1 rev1=1 rx=0x7f83380628c0 tx=0x7f833c037000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:15.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f833801cbc0 2021-03-09T09:48:15.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f833801cbc0 2021-03-09T09:48:15.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:15.288+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01d760 con 0x7f833801cbc0 2021-03-09T09:48:15.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:48:15 smithi146 bash[13136]: debug 2021-03-09T09:48:15.282+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13736 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:48:20.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:48:20.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:20.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380100b0 secure :-1 s=READY pgs=4832 cs=0 l=1 rev1=1 rx=0x7f83380628c0 tx=0x7f833c037000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:20.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380100b0 secure :-1 s=READY pgs=4832 cs=0 l=1 rev1=1 rx=0x7f83380628c0 tx=0x7f833c037000).stop 2021-03-09T09:48:20.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f83380100b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:20.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380100b0 unknown :-1 s=CLOSED pgs=4832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:20.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:20.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:20.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:20.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:48:20.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c300 con 0x7f833802d7b0 2021-03-09T09:48:20.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010e60 con 0x7f8324005e00 2021-03-09T09:48:20.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:48:20.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:20.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:20.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:48:20.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:20.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:20.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:20.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:20.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:20.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.284+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338021970 con 0x7f833802d7b0 2021-03-09T09:48:20.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.288+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4842 cs=0 l=1 rev1=1 rx=0x7f833c0095f0 tx=0x7f833c037000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:20.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.288+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c065ed0 con 0x7f833802d7b0 2021-03-09T09:48:20.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.288+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079e60 con 0x7f833802d7b0 2021-03-09T09:48:20.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:20.292+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f833802d7b0 2021-03-09T09:48:20.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:48:20 smithi146 bash[13136]: debug 2021-03-09T09:48:20.286+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13763 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:48:25.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.284+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:48:25.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.284+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:25.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.284+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4842 cs=0 l=1 rev1=1 rx=0x7f833c0095f0 tx=0x7f833c037000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:25.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.284+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4842 cs=0 l=1 rev1=1 rx=0x7f833c0095f0 tx=0x7f833c037000).stop 2021-03-09T09:48:25.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:25.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.284+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:25.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.284+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:25.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.284+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:25.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.284+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:25.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380614c0 con 0x7f8324003400 2021-03-09T09:48:25.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e0c0 con 0x7f833801cbc0 2021-03-09T09:48:25.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.284+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b7d0 con 0x7f8324005e00 2021-03-09T09:48:25.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.288+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:25.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.288+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:48:25.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.288+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:25.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.288+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:48:25.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.288+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:25.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.296+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:25.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.296+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:25.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.296+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:25.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.296+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:25.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.296+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806ce00 con 0x7f833801cbc0 2021-03-09T09:48:25.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.296+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 secure :-1 s=READY pgs=4853 cs=0 l=1 rev1=1 rx=0x7f8344054ec0 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:25.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.296+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833801cbc0 2021-03-09T09:48:25.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.300+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833801cbc0 2021-03-09T09:48:25.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:25.300+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833801cbc0 2021-03-09T09:48:25.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:48:25 smithi146 bash[13136]: debug 2021-03-09T09:48:25.290+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13790 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:48:30.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:48:30.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:30.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 secure :-1 s=READY pgs=4853 cs=0 l=1 rev1=1 rx=0x7f8344054ec0 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:30.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 secure :-1 s=READY pgs=4853 cs=0 l=1 rev1=1 rx=0x7f8344054ec0 tx=0x7f8344049b10).stop 2021-03-09T09:48:30.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:30.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 unknown :-1 s=CLOSED pgs=4853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:30.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:30.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:30.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:48:30.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:30.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:30.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014690 con 0x7f8324003400 2021-03-09T09:48:30.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a990 con 0x7f833802d7b0 2021-03-09T09:48:30.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f0a0 con 0x7f8324005e00 2021-03-09T09:48:30.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:30.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:48:30.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:30.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:30.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:30.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:30.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:30.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012780 con 0x7f833802d7b0 2021-03-09T09:48:30.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.292+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4861 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c037000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:30.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.296+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c065ed0 con 0x7f833802d7b0 2021-03-09T09:48:30.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.296+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c018620 con 0x7f833802d7b0 2021-03-09T09:48:30.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:30.300+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833802d7b0 2021-03-09T09:48:30.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:48:30 smithi146 bash[13136]: debug 2021-03-09T09:48:30.294+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13814 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:48:35.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:48:35.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:35.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4861 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c037000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:35.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4861 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c037000).stop 2021-03-09T09:48:35.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:35.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:35.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:35.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:35.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:35.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380680e0 con 0x7f8324003400 2021-03-09T09:48:35.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f833801cbc0 2021-03-09T09:48:35.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380614c0 con 0x7f8324005e00 2021-03-09T09:48:35.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:48:35.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:35.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:48:35.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:35.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.295+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:35.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.299+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:35.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.299+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:35.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.299+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:35.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.299+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:35.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.299+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338003470 con 0x7f833801cbc0 2021-03-09T09:48:35.309 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.299+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 secure :-1 s=READY pgs=4868 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c037000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:35.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.299+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01c4d0 con 0x7f833801cbc0 2021-03-09T09:48:35.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.299+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c018620 con 0x7f833801cbc0 2021-03-09T09:48:35.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:35.307+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c061370 con 0x7f833801cbc0 2021-03-09T09:48:35.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:48:35 smithi146 bash[13136]: debug 2021-03-09T09:48:35.294+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13841 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:48:40.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.295+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:48:40.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.295+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:40.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.295+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 secure :-1 s=READY pgs=4868 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c037000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:40.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.295+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 secure :-1 s=READY pgs=4868 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c037000).stop 2021-03-09T09:48:40.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.295+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:40.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.295+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 unknown :-1 s=CLOSED pgs=4868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:40.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.295+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:40.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.295+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:40.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.295+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:40.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.295+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b7d0 con 0x7f8324003400 2021-03-09T09:48:40.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.295+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a990 con 0x7f833802d7b0 2021-03-09T09:48:40.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.295+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f0a0 con 0x7f8324005e00 2021-03-09T09:48:40.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.299+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:48:40.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.299+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:48:40.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.299+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:40.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.299+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:40.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.299+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:40.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.299+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83380049c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:40.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.299+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83380049c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:40.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.299+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:40.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.299+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:40.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.299+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801d6c0 con 0x7f833802d7b0 2021-03-09T09:48:40.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.303+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4877 cs=0 l=1 rev1=1 rx=0x7f833806de90 tx=0x7f833c037000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:40.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.303+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018620 con 0x7f833802d7b0 2021-03-09T09:48:40.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.303+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05c7f0 con 0x7f833802d7b0 2021-03-09T09:48:40.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:40.307+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c061300 con 0x7f833802d7b0 2021-03-09T09:48:40.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:48:40 smithi146 bash[13136]: debug 2021-03-09T09:48:40.298+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13863 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:48:45.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:48:45.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:45.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4877 cs=0 l=1 rev1=1 rx=0x7f833806de90 tx=0x7f833c037000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:45.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 secure :-1 s=READY pgs=4877 cs=0 l=1 rev1=1 rx=0x7f833806de90 tx=0x7f833c037000).stop 2021-03-09T09:48:45.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:45.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=4877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:45.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:45.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:45.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:45.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a1b0 con 0x7f8324003400 2021-03-09T09:48:45.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fda0 con 0x7f833801cbc0 2021-03-09T09:48:45.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:48:45.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:48:45.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:45.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:45.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:48:45.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.299+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:45.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.303+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:45.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.303+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:45.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.303+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:45.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.303+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:45.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.303+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e070 con 0x7f833801cbc0 2021-03-09T09:48:45.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.303+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 secure :-1 s=READY pgs=4884 cs=0 l=1 rev1=1 rx=0x7f833c007a40 tx=0x7f833c053530).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:45.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.303+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c007880 con 0x7f833801cbc0 2021-03-09T09:48:45.309 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.303+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c007470 con 0x7f833801cbc0 2021-03-09T09:48:45.309 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:45.307+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02ec30 con 0x7f833801cbc0 2021-03-09T09:48:45.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:48:45 smithi146 bash[13136]: debug 2021-03-09T09:48:45.302+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13889 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:48:50.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:48:50.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:50.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 secure :-1 s=READY pgs=4884 cs=0 l=1 rev1=1 rx=0x7f833c007a40 tx=0x7f833c053530).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:50.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 secure :-1 s=READY pgs=4884 cs=0 l=1 rev1=1 rx=0x7f833c007a40 tx=0x7f833c053530).stop 2021-03-09T09:48:50.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f8338007990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:50.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f8338007990 unknown :-1 s=CLOSED pgs=4884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:50.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380299a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:50.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:50.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:50.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022dc0 con 0x7f8324003400 2021-03-09T09:48:50.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069860 con 0x7f833802d7b0 2021-03-09T09:48:50.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a070 con 0x7f8324005e00 2021-03-09T09:48:50.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:48:50.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:50.309 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:48:50.309 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:50.309 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380299a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:50.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:50.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:50.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:50.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:50.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380116b0 con 0x7f833802d7b0 2021-03-09T09:48:50.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.303+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380299a0 secure :-1 s=READY pgs=4893 cs=0 l=1 rev1=1 rx=0x7f833c06b040 tx=0x7f833c053530).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:50.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.307+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c007650 con 0x7f833802d7b0 2021-03-09T09:48:50.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.307+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0912d0 con 0x7f833802d7b0 2021-03-09T09:48:50.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:50.311+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c061300 con 0x7f833802d7b0 2021-03-09T09:48:50.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:48:50 smithi146 bash[13136]: debug 2021-03-09T09:48:50.302+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13913 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:48:55.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380299a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:48:55.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380299a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:55.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380299a0 secure :-1 s=READY pgs=4893 cs=0 l=1 rev1=1 rx=0x7f833c06b040 tx=0x7f833c053530).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:55.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380299a0 secure :-1 s=READY pgs=4893 cs=0 l=1 rev1=1 rx=0x7f833c06b040 tx=0x7f833c053530).stop 2021-03-09T09:48:55.309 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380299a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:55.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380299a0 unknown :-1 s=CLOSED pgs=4893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:55.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:55.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380082f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:55.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:55.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:48:55.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800bd60 con 0x7f833801cbc0 2021-03-09T09:48:55.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061130 con 0x7f8324005e00 2021-03-09T09:48:55.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:48:55.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:55.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:48:55.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:55.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:55.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:55.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:55.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:55.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:55.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012780 con 0x7f833801cbc0 2021-03-09T09:48:55.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.307+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380082f0 secure :-1 s=READY pgs=4902 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:55.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.311+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833801cbc0 2021-03-09T09:48:55.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.311+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833801cbc0 2021-03-09T09:48:55.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:48:55.311+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ac40 con 0x7f833801cbc0 2021-03-09T09:48:55.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:48:55 smithi146 bash[13136]: debug 2021-03-09T09:48:55.306+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13943 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:49:00.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.307+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f83380082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:49:00.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.307+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f83380082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:00.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.307+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380082f0 secure :-1 s=READY pgs=4902 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:00.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.307+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380082f0 secure :-1 s=READY pgs=4902 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344049b10).stop 2021-03-09T09:49:00.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.307+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f83380082f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:00.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.307+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380082f0 unknown :-1 s=CLOSED pgs=4902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:00.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.307+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338063080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:00.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.307+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:00.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.307+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:00.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.307+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005830 con 0x7f8324003400 2021-03-09T09:49:00.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.307+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068e50 con 0x7f833802d7b0 2021-03-09T09:49:00.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.307+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8324005e00 2021-03-09T09:49:00.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.307+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:49:00.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.307+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:00.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.307+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:49:00.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.307+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:00.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.311+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338063080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:00.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.311+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:00.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.311+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:00.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.311+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:00.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.311+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:00.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.311+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e070 con 0x7f833802d7b0 2021-03-09T09:49:00.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.311+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338063080 secure :-1 s=READY pgs=4910 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c077000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:00.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.315+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0218d0 con 0x7f833802d7b0 2021-03-09T09:49:00.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.315+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c4d0 con 0x7f833802d7b0 2021-03-09T09:49:00.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:00.319+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c061300 con 0x7f833802d7b0 2021-03-09T09:49:00.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:49:00 smithi146 bash[13136]: debug 2021-03-09T09:49:00.310+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13968 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:49:05.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:49:05.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:05.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338063080 secure :-1 s=READY pgs=4910 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c077000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:05.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338063080 secure :-1 s=READY pgs=4910 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c077000).stop 2021-03-09T09:49:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338063080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338063080 unknown :-1 s=CLOSED pgs=4910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833800ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:05.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:49:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d860 con 0x7f833801cbc0 2021-03-09T09:49:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b240 con 0x7f8324005e00 2021-03-09T09:49:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:49:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:49:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.311+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833800ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.315+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.315+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.315+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.315+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.315+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026af0 con 0x7f833801cbc0 2021-03-09T09:49:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.315+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833800ce20 secure :-1 s=READY pgs=4919 cs=0 l=1 rev1=1 rx=0x7f8338007770 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.319+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344032930 con 0x7f833801cbc0 2021-03-09T09:49:05.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.319+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833801cbc0 2021-03-09T09:49:05.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:05.323+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440791e0 con 0x7f833801cbc0 2021-03-09T09:49:05.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:49:05 smithi146 bash[13136]: debug 2021-03-09T09:49:05.309+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 13991 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:49:10.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833800ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:49:10.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833800ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:10.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833800ce20 secure :-1 s=READY pgs=4919 cs=0 l=1 rev1=1 rx=0x7f8338007770 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:10.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833800ce20 secure :-1 s=READY pgs=4919 cs=0 l=1 rev1=1 rx=0x7f8338007770 tx=0x7f8344049b10).stop 2021-03-09T09:49:10.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833800ce20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:10.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833800ce20 unknown :-1 s=CLOSED pgs=4919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:10.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338063080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:10.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:10.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:10.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cbd0 con 0x7f8324003400 2021-03-09T09:49:10.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007050 con 0x7f833802d7b0 2021-03-09T09:49:10.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8324005e00 2021-03-09T09:49:10.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:49:10.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:10.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:49:10.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:10.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.315+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338063080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:10.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.319+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:10.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.319+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:10.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.319+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:10.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.319+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:10.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.319+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380256a0 con 0x7f833802d7b0 2021-03-09T09:49:10.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.319+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338063080 secure :-1 s=READY pgs=4933 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c077000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:10.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.323+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01c4d0 con 0x7f833802d7b0 2021-03-09T09:49:10.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.323+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c042ba0 con 0x7f833802d7b0 2021-03-09T09:49:10.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:10.323+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c061300 con 0x7f833802d7b0 2021-03-09T09:49:10.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:49:10 smithi146 bash[13136]: debug 2021-03-09T09:49:10.313+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14019 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:49:15.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.315+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:49:15.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.315+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:15.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.315+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338063080 secure :-1 s=READY pgs=4933 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c077000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:15.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.315+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338063080 secure :-1 s=READY pgs=4933 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c077000).stop 2021-03-09T09:49:15.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.315+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338063080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:15.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.315+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338063080 unknown :-1 s=CLOSED pgs=4933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:15.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.315+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:15.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.315+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:15.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.315+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:15.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.315+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:49:15.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.315+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068e50 con 0x7f833801cbc0 2021-03-09T09:49:15.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.315+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005830 con 0x7f8324005e00 2021-03-09T09:49:15.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.315+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:49:15.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.315+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:15.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.319+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800ce20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:49:15.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.319+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800ce20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:15.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.319+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:15.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.319+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:15.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.319+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:15.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.319+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800ce20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:15.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.319+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800ce20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:15.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.319+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806de40 con 0x7f833801cbc0 2021-03-09T09:49:15.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.319+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=4936 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c077000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:15.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.319+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01c4d0 con 0x7f833801cbc0 2021-03-09T09:49:15.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.319+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c018620 con 0x7f833801cbc0 2021-03-09T09:49:15.333 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:15.323+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f833801cbc0 2021-03-09T09:49:15.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:49:15 smithi146 bash[13136]: debug 2021-03-09T09:49:15.317+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14045 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:49:20.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:49:20.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:20.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=4936 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c077000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:20.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=4936 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c077000).stop 2021-03-09T09:49:20.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:20.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=4936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:20.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380299a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:20.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:20.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800ce20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:49:20.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800ce20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:20.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:20.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d040 con 0x7f8324003400 2021-03-09T09:49:20.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007050 con 0x7f833802d7b0 2021-03-09T09:49:20.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8324005e00 2021-03-09T09:49:20.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380299a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:20.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:49:20.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.318+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:20.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.322+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:20.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.322+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:20.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.322+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800ce20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:20.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.322+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800ce20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:20.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.322+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd50 con 0x7f833802d7b0 2021-03-09T09:49:20.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.322+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380299a0 secure :-1 s=READY pgs=4945 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c077000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:20.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.322+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01c4d0 con 0x7f833802d7b0 2021-03-09T09:49:20.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.322+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c018620 con 0x7f833802d7b0 2021-03-09T09:49:20.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:20.326+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f833802d7b0 2021-03-09T09:49:20.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:49:20 smithi146 bash[13136]: debug 2021-03-09T09:49:20.317+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14059 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:49:25.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380299a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:49:25.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380299a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:25.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380299a0 secure :-1 s=READY pgs=4945 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c077000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:25.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380299a0 secure :-1 s=READY pgs=4945 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c077000).stop 2021-03-09T09:49:25.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380299a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:25.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380299a0 unknown :-1 s=CLOSED pgs=4945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:25.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:25.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:25.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:25.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:49:25.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d9f0 con 0x7f833801cbc0 2021-03-09T09:49:25.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380682d0 con 0x7f8324005e00 2021-03-09T09:49:25.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:49:25.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:25.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:25.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800ce20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:49:25.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800ce20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:25.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:25.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:25.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800ce20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:25.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800ce20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:25.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd50 con 0x7f833801cbc0 2021-03-09T09:49:25.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.322+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=4954 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f834400f220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:25.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.326+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833801cbc0 2021-03-09T09:49:25.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.326+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833801cbc0 2021-03-09T09:49:25.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:25.326+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344072190 con 0x7f833801cbc0 2021-03-09T09:49:25.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:49:25 smithi146 bash[13136]: debug 2021-03-09T09:49:25.321+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14093 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:49:30.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:49:30.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:30.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=4954 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f834400f220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:30.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=4954 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f834400f220).stop 2021-03-09T09:49:30.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:30.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=4954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:30.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338061aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:30.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:30.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:30.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800bd60 con 0x7f8324003400 2021-03-09T09:49:30.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380107b0 con 0x7f833802d7b0 2021-03-09T09:49:30.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cf60 con 0x7f8324005e00 2021-03-09T09:49:30.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cdd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:49:30.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:30.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:49:30.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338061aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:30.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.322+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:30.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.326+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:30.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.326+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:30.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.326+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cdd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:30.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.326+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:30.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.326+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338061f90 con 0x7f833802d7b0 2021-03-09T09:49:30.333 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.326+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338061aa0 secure :-1 s=READY pgs=4962 cs=0 l=1 rev1=1 rx=0x7f833c011000 tx=0x7f833c077000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:30.333 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.326+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c06b6b0 con 0x7f833802d7b0 2021-03-09T09:49:30.333 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.326+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01d710 con 0x7f833802d7b0 2021-03-09T09:49:30.334 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:30.330+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06b370 con 0x7f833802d7b0 2021-03-09T09:49:30.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:49:30 smithi146 bash[13136]: debug 2021-03-09T09:49:30.325+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14110 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:49:35.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338061aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:49:35.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338061aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:35.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338061aa0 secure :-1 s=READY pgs=4962 cs=0 l=1 rev1=1 rx=0x7f833c011000 tx=0x7f833c077000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:35.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338061aa0 secure :-1 s=READY pgs=4962 cs=0 l=1 rev1=1 rx=0x7f833c011000 tx=0x7f833c077000).stop 2021-03-09T09:49:35.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338061aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:35.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338061aa0 unknown :-1 s=CLOSED pgs=4962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:35.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:35.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:35.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:35.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8324003400 2021-03-09T09:49:35.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006930 con 0x7f833801cbc0 2021-03-09T09:49:35.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:49:35.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:49:35.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:35.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:35.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cdd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:49:35.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.326+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:35.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.330+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:35.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.330+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:35.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.330+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cdd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:35.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.330+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:35.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.330+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338016240 con 0x7f833801cbc0 2021-03-09T09:49:35.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.330+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=4972 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834400f220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:35.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.330+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344078190 con 0x7f833801cbc0 2021-03-09T09:49:35.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.330+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033040 con 0x7f833801cbc0 2021-03-09T09:49:35.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:35.334+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344049740 con 0x7f833801cbc0 2021-03-09T09:49:35.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:49:35 smithi146 bash[13136]: debug 2021-03-09T09:49:35.325+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14136 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:49:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:49:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=4972 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834400f220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:40.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=4972 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834400f220).stop 2021-03-09T09:49:40.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:40.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=4972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:40.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338014870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:40.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:40.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:40.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026720 con 0x7f8324003400 2021-03-09T09:49:40.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d310 con 0x7f833802d7b0 2021-03-09T09:49:40.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cf60 con 0x7f8324005e00 2021-03-09T09:49:40.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:49:40.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:40.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338014870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:40.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cdd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:49:40.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.330+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:40.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.338+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:40.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.338+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:40.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.338+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cdd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:40.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.338+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:40.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.338+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c290 con 0x7f833802d7b0 2021-03-09T09:49:40.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.338+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338014870 secure :-1 s=READY pgs=4982 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c077000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:40.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.338+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01d710 con 0x7f833802d7b0 2021-03-09T09:49:40.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.338+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c4d0 con 0x7f833802d7b0 2021-03-09T09:49:40.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:40.342+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06b370 con 0x7f833802d7b0 2021-03-09T09:49:40.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:49:40 smithi146 bash[13136]: debug 2021-03-09T09:49:40.329+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14168 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:49:45.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.330+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338014870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:49:45.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.330+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338014870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:45.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.330+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338014870 secure :-1 s=READY pgs=4982 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c077000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:45.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.330+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338014870 secure :-1 s=READY pgs=4982 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c077000).stop 2021-03-09T09:49:45.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.330+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338014870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:45.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.330+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338014870 unknown :-1 s=CLOSED pgs=4982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:45.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.330+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:45.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.330+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:45.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.330+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:45.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.334+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d9f0 con 0x7f8324003400 2021-03-09T09:49:45.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.334+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f833801cbc0 2021-03-09T09:49:45.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.334+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028d40 con 0x7f8324005e00 2021-03-09T09:49:45.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.334+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:49:45.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.334+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:45.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.334+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cdd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:49:45.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.334+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:45.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.334+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:45.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.334+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:45.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.334+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:45.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.334+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cdd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:45.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.334+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:45.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.334+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd50 con 0x7f833801cbc0 2021-03-09T09:49:45.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.338+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=4988 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834400f220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:45.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.338+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401f950 con 0x7f833801cbc0 2021-03-09T09:49:45.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.338+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833801cbc0 2021-03-09T09:49:45.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:45.338+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496f0 con 0x7f833801cbc0 2021-03-09T09:49:45.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:49:45 smithi146 bash[13136]: debug 2021-03-09T09:49:45.333+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14188 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:49:50.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.334+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:49:50.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.334+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:50.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.334+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=4988 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834400f220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:50.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.334+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=4988 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834400f220).stop 2021-03-09T09:49:50.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.334+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:50.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.334+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=4988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:50.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.334+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338015160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:50.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.334+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:50.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.334+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:50.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.334+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8324003400 2021-03-09T09:49:50.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.334+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d310 con 0x7f833802d7b0 2021-03-09T09:49:50.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.334+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cf60 con 0x7f8324005e00 2021-03-09T09:49:50.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.334+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cdd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:49:50.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.334+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:50.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.338+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338015160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:50.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.338+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:49:50.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.338+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:50.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.338+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:50.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.338+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:50.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.338+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cdd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:50.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.338+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:50.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.338+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338066700 con 0x7f833802d7b0 2021-03-09T09:49:50.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.338+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338015160 secure :-1 s=READY pgs=4998 cs=0 l=1 rev1=1 rx=0x7f833c041640 tx=0x7f833c077000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:50.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.342+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01d710 con 0x7f833802d7b0 2021-03-09T09:49:50.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.342+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0416b0 con 0x7f833802d7b0 2021-03-09T09:49:50.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:50.346+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01d3a0 con 0x7f833802d7b0 2021-03-09T09:49:50.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:49:50 smithi146 bash[13136]: debug 2021-03-09T09:49:50.337+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14215 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:49:55.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338015160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:49:55.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338015160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:55.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338015160 secure :-1 s=READY pgs=4998 cs=0 l=1 rev1=1 rx=0x7f833c041640 tx=0x7f833c077000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:55.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338015160 secure :-1 s=READY pgs=4998 cs=0 l=1 rev1=1 rx=0x7f833c041640 tx=0x7f833c077000).stop 2021-03-09T09:49:55.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338015160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:55.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338015160 unknown :-1 s=CLOSED pgs=4998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:55.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:55.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:55.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:55.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006930 con 0x7f8324003400 2021-03-09T09:49:55.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f833801cbc0 2021-03-09T09:49:55.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:49:55.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:49:55.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:49:55.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:55.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:55.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:55.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:55.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:55.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:55.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:55.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801da70 con 0x7f833801cbc0 2021-03-09T09:49:55.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.338+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5006 cs=0 l=1 rev1=1 rx=0x7f833c05ca60 tx=0x7f833c077000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:55.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.342+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0296d0 con 0x7f833801cbc0 2021-03-09T09:49:55.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.342+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0294a0 con 0x7f833801cbc0 2021-03-09T09:49:55.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:49:55.346+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833801cbc0 2021-03-09T09:49:55.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:49:55 smithi146 bash[13136]: debug 2021-03-09T09:49:55.337+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14241 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:50:00.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:50:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5006 cs=0 l=1 rev1=1 rx=0x7f833c05ca60 tx=0x7f833c077000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5006 cs=0 l=1 rev1=1 rx=0x7f833c05ca60 tx=0x7f833c077000).stop 2021-03-09T09:50:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338029960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002f00 con 0x7f8324003400 2021-03-09T09:50:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f833802d7b0 2021-03-09T09:50:00.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e080 con 0x7f8324005e00 2021-03-09T09:50:00.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:50:00.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:00.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338029960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:00.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:50:00.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.342+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:00.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.346+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:00.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.346+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:00.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.346+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:00.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.346+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:00.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.346+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806d9a0 con 0x7f833802d7b0 2021-03-09T09:50:00.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.346+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338029960 secure :-1 s=READY pgs=5014 cs=0 l=1 rev1=1 rx=0x7f833c061ff0 tx=0x7f833c077000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:00.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.346+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00f670 con 0x7f833802d7b0 2021-03-09T09:50:00.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.346+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00f670 con 0x7f833802d7b0 2021-03-09T09:50:00.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:00.350+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054d80 con 0x7f833802d7b0 2021-03-09T09:50:00.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:50:00 smithi146 bash[13136]: debug 2021-03-09T09:50:00.341+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14274 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:50:05.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338029960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:50:05.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338029960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:05.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338029960 secure :-1 s=READY pgs=5014 cs=0 l=1 rev1=1 rx=0x7f833c061ff0 tx=0x7f833c077000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:05.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338029960 secure :-1 s=READY pgs=5014 cs=0 l=1 rev1=1 rx=0x7f833c061ff0 tx=0x7f833c077000).stop 2021-03-09T09:50:05.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338029960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:05.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338029960 unknown :-1 s=CLOSED pgs=5014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:05.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:05.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:05.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:05.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380067f0 con 0x7f8324003400 2021-03-09T09:50:05.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d310 con 0x7f833801cbc0 2021-03-09T09:50:05.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cf60 con 0x7f8324005e00 2021-03-09T09:50:05.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:50:05.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:05.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:50:05.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:05.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:05.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:05.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:05.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:05.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:05.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c290 con 0x7f833801cbc0 2021-03-09T09:50:05.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5023 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f834400f220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:05.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344054d90 con 0x7f833801cbc0 2021-03-09T09:50:05.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.346+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833801cbc0 2021-03-09T09:50:05.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:05.354+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833801cbc0 2021-03-09T09:50:05.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:50:05 smithi146 bash[13136]: debug 2021-03-09T09:50:05.344+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14299 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:50:10.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:50:10.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:10.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5023 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f834400f220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:10.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5023 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f834400f220).stop 2021-03-09T09:50:10.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:10.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:10.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:10.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:10.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338015d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:10.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:50:10.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800bd60 con 0x7f833802d7b0 2021-03-09T09:50:10.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022dc0 con 0x7f8324005e00 2021-03-09T09:50:10.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:50:10.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:10.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:50:10.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:10.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338015d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:10.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:10.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:10.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:10.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:10.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005c60 con 0x7f833802d7b0 2021-03-09T09:50:10.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.349+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338015d60 secure :-1 s=READY pgs=5032 cs=0 l=1 rev1=1 rx=0x7f833c053c60 tx=0x7f833c077000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:10.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c053f00 con 0x7f833802d7b0 2021-03-09T09:50:10.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c053f00 con 0x7f833802d7b0 2021-03-09T09:50:10.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:10.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0412e0 con 0x7f833802d7b0 2021-03-09T09:50:10.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:50:10 smithi146 bash[13136]: debug 2021-03-09T09:50:10.348+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14318 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:50:15.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.349+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338015d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:50:15.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.349+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338015d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:15.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.349+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338015d60 secure :-1 s=READY pgs=5032 cs=0 l=1 rev1=1 rx=0x7f833c053c60 tx=0x7f833c077000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:15.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.349+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338015d60 secure :-1 s=READY pgs=5032 cs=0 l=1 rev1=1 rx=0x7f833c053c60 tx=0x7f833c077000).stop 2021-03-09T09:50:15.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338015d60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:15.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.349+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338015d60 unknown :-1 s=CLOSED pgs=5032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:15.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.349+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380252a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:15.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.349+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:15.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.349+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:15.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338070100 con 0x7f8324003400 2021-03-09T09:50:15.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068e50 con 0x7f833801cbc0 2021-03-09T09:50:15.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.349+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:50:15.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063f00 con 0x7f8324005e00 2021-03-09T09:50:15.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.349+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:15.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.349+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380252a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:15.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:50:15.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:15.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:15.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:15.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:15.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:15.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806ff70 con 0x7f833801cbc0 2021-03-09T09:50:15.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380252a0 secure :-1 s=READY pgs=5041 cs=0 l=1 rev1=1 rx=0x7f833c057450 tx=0x7f833c077000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:15.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c053cd0 con 0x7f833801cbc0 2021-03-09T09:50:15.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c053cd0 con 0x7f833801cbc0 2021-03-09T09:50:15.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:15.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f833801cbc0 2021-03-09T09:50:15.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:50:15 smithi146 bash[13136]: debug 2021-03-09T09:50:15.348+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14348 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:50:20.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f83380252a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:50:20.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f83380252a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:20.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380252a0 secure :-1 s=READY pgs=5041 cs=0 l=1 rev1=1 rx=0x7f833c057450 tx=0x7f833c077000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:20.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380252a0 secure :-1 s=READY pgs=5041 cs=0 l=1 rev1=1 rx=0x7f833c057450 tx=0x7f833c077000).stop 2021-03-09T09:50:20.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f83380252a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:20.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f83380252a0 unknown :-1 s=CLOSED pgs=5041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:20.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:20.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:20.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:20.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8324003400 2021-03-09T09:50:20.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f833802d7b0 2021-03-09T09:50:20.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:50:20.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:50:20.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:20.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:50:20.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:20.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:20.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:20.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:20.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800cad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:20.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800cad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:20.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ed40 con 0x7f833802d7b0 2021-03-09T09:50:20.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.357+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 secure :-1 s=READY pgs=5051 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c077000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:20.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05c9d0 con 0x7f833802d7b0 2021-03-09T09:50:20.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05cbe0 con 0x7f833802d7b0 2021-03-09T09:50:20.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:20.361+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833802d7b0 2021-03-09T09:50:20.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:50:20 smithi146 bash[13136]: debug 2021-03-09T09:50:20.352+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14369 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:50:25.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:50:25.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:25.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 secure :-1 s=READY pgs=5051 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c077000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:25.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 secure :-1 s=READY pgs=5051 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c077000).stop 2021-03-09T09:50:25.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833800f2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:25.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 unknown :-1 s=CLOSED pgs=5051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:25.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:25.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:25.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:25.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063f00 con 0x7f8324003400 2021-03-09T09:50:25.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f833801cbc0 2021-03-09T09:50:25.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002520 con 0x7f8324005e00 2021-03-09T09:50:25.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:50:25.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:25.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:50:25.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:25.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:25.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.357+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:25.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.357+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:25.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.357+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:25.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.357+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:25.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.357+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380690f0 con 0x7f833801cbc0 2021-03-09T09:50:25.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.357+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5058 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c01c5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:25.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c06e8b0 con 0x7f833801cbc0 2021-03-09T09:50:25.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c06e680 con 0x7f833801cbc0 2021-03-09T09:50:25.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:25.361+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06eae0 con 0x7f833801cbc0 2021-03-09T09:50:25.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:50:25 smithi146 bash[13136]: debug 2021-03-09T09:50:25.356+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14400 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:50:30.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:50:30.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:30.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5058 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c01c5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:30.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5058 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c01c5d0).stop 2021-03-09T09:50:30.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:30.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:30.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:30.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:30.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:30.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c830 con 0x7f8324003400 2021-03-09T09:50:30.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d860 con 0x7f833802d7b0 2021-03-09T09:50:30.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380157e0 con 0x7f8324005e00 2021-03-09T09:50:30.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:50:30.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:30.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:50:30.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:30.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:30.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:30.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:30.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:30.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:30.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.357+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024710 con 0x7f833802d7b0 2021-03-09T09:50:30.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.361+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 secure :-1 s=READY pgs=5066 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f834400f220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:30.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.361+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833802d7b0 2021-03-09T09:50:30.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.361+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f833802d7b0 2021-03-09T09:50:30.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:30.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834401f120 con 0x7f833802d7b0 2021-03-09T09:50:30.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:50:30 smithi146 bash[13136]: debug 2021-03-09T09:50:30.360+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14427 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:50:35.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.357+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:50:35.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.357+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:35.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.357+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 secure :-1 s=READY pgs=5066 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f834400f220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:35.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 secure :-1 s=READY pgs=5066 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f834400f220).stop 2021-03-09T09:50:35.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833800f2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:35.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 unknown :-1 s=CLOSED pgs=5066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:35.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:35.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:35.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:35.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8324003400 2021-03-09T09:50:35.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f833801cbc0 2021-03-09T09:50:35.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:50:35.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:50:35.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:35.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:50:35.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:35.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:35.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:35.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:35.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:35.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:35.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.361+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380107b0 con 0x7f833801cbc0 2021-03-09T09:50:35.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5075 cs=0 l=1 rev1=1 rx=0x7f833c05f7d0 tx=0x7f833c01c5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:35.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05f840 con 0x7f833801cbc0 2021-03-09T09:50:35.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c018620 con 0x7f833801cbc0 2021-03-09T09:50:35.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:35.369+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c042ce0 con 0x7f833801cbc0 2021-03-09T09:50:35.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:50:35 smithi146 bash[13136]: debug 2021-03-09T09:50:35.360+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14451 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:50:40.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:50:40.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:40.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5075 cs=0 l=1 rev1=1 rx=0x7f833c05f7d0 tx=0x7f833c01c5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:40.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5075 cs=0 l=1 rev1=1 rx=0x7f833c05f7d0 tx=0x7f833c01c5d0).stop 2021-03-09T09:50:40.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:40.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:40.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338006030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:40.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:40.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:40.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b440 con 0x7f8324003400 2021-03-09T09:50:40.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f833802d7b0 2021-03-09T09:50:40.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061130 con 0x7f8324005e00 2021-03-09T09:50:40.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:50:40.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:50:40.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:40.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:40.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:40.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:40.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:40.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:40.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:40.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380214d0 con 0x7f833802d7b0 2021-03-09T09:50:40.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338006030 secure :-1 s=READY pgs=5084 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c01c5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:40.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.369+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01d3a0 con 0x7f833802d7b0 2021-03-09T09:50:40.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.369+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01d500 con 0x7f833802d7b0 2021-03-09T09:50:40.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:40.373+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f833802d7b0 2021-03-09T09:50:40.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:50:40 smithi146 bash[13136]: debug 2021-03-09T09:50:40.364+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14478 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:50:45.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.365+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338006030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:50:45.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.365+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338006030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:45.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338006030 secure :-1 s=READY pgs=5084 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c01c5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:45.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338006030 secure :-1 s=READY pgs=5084 cs=0 l=1 rev1=1 rx=0x7f833c04ded0 tx=0x7f833c01c5d0).stop 2021-03-09T09:50:45.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338006030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:45.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.365+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338006030 unknown :-1 s=CLOSED pgs=5084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:45.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.365+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:45.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.365+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:45.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.365+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:45.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380635e0 con 0x7f8324003400 2021-03-09T09:50:45.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380199f0 con 0x7f833801cbc0 2021-03-09T09:50:45.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8324005e00 2021-03-09T09:50:45.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.365+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:50:45.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.365+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:45.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.365+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:50:45.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:45.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.369+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:45.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.369+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:45.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.369+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:45.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.369+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:45.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.369+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:45.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.369+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017550 con 0x7f833801cbc0 2021-03-09T09:50:45.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.369+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5092 cs=0 l=1 rev1=1 rx=0x7f833c009980 tx=0x7f833c01c5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:45.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.373+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0572b0 con 0x7f833801cbc0 2021-03-09T09:50:45.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.373+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0574b0 con 0x7f833801cbc0 2021-03-09T09:50:45.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:45.377+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f833801cbc0 2021-03-09T09:50:45.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:50:45 smithi146 bash[13136]: debug 2021-03-09T09:50:45.368+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14501 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:50:50.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:50:50.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:50.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5092 cs=0 l=1 rev1=1 rx=0x7f833c009980 tx=0x7f833c01c5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:50.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5092 cs=0 l=1 rev1=1 rx=0x7f833c009980 tx=0x7f833c01c5d0).stop 2021-03-09T09:50:50.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:50.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:50.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:50.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:50.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:50.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8324003400 2021-03-09T09:50:50.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f833802d7b0 2021-03-09T09:50:50.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006550 con 0x7f8324005e00 2021-03-09T09:50:50.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:50:50.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:50:50.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:50.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:50.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.369+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:50.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.373+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:50.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.373+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:50.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.373+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:50.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.373+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:50.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.373+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c290 con 0x7f833802d7b0 2021-03-09T09:50:50.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.373+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 secure :-1 s=READY pgs=5102 cs=0 l=1 rev1=1 rx=0x7f833c01c640 tx=0x7f833c01c5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:50.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.377+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0912d0 con 0x7f833802d7b0 2021-03-09T09:50:50.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.377+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c054ed0 con 0x7f833802d7b0 2021-03-09T09:50:50.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:50.381+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833802d7b0 2021-03-09T09:50:50.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:50:50 smithi146 bash[13136]: debug 2021-03-09T09:50:50.372+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14529 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:50:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:50:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 secure :-1 s=READY pgs=5102 cs=0 l=1 rev1=1 rx=0x7f833c01c640 tx=0x7f833c01c5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 secure :-1 s=READY pgs=5102 cs=0 l=1 rev1=1 rx=0x7f833c01c640 tx=0x7f833c01c5d0).stop 2021-03-09T09:50:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f833800f2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f833800f2b0 unknown :-1 s=CLOSED pgs=5102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8324003400 2021-03-09T09:50:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f833801cbc0 2021-03-09T09:50:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026720 con 0x7f8324005e00 2021-03-09T09:50:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:50:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:50:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.372+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.376+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.376+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.376+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.376+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.376+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380214d0 con 0x7f833801cbc0 2021-03-09T09:50:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.376+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5110 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834400f220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.376+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833801cbc0 2021-03-09T09:50:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.376+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833801cbc0 2021-03-09T09:50:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:50:55.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440321b0 con 0x7f833801cbc0 2021-03-09T09:50:55.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:50:55 smithi146 bash[13136]: debug 2021-03-09T09:50:55.376+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14553 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:51:00.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.372+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:51:00.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.372+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:00.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.372+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5110 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834400f220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:00.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.372+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5110 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834400f220).stop 2021-03-09T09:51:00.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.372+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:00.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.372+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:00.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.372+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:00.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.372+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:00.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.372+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:00.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.372+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006550 con 0x7f8324003400 2021-03-09T09:51:00.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.372+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f833802d7b0 2021-03-09T09:51:00.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.372+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a480 con 0x7f8324005e00 2021-03-09T09:51:00.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.372+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:51:00.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.372+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:00.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.376+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:51:00.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.376+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:00.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.376+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:00.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.376+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:00.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.376+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:00.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.376+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:00.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.376+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:00.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.376+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f833802d7b0 2021-03-09T09:51:00.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.376+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 secure :-1 s=READY pgs=5118 cs=0 l=1 rev1=1 rx=0x7f834404c690 tx=0x7f834400f220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:00.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401e910 con 0x7f833802d7b0 2021-03-09T09:51:00.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f833802d7b0 2021-03-09T09:51:00.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:00.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344032140 con 0x7f833802d7b0 2021-03-09T09:51:00.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:51:00 smithi146 bash[13136]: debug 2021-03-09T09:51:00.379+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14580 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:51:05.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:51:05.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:05.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 secure :-1 s=READY pgs=5118 cs=0 l=1 rev1=1 rx=0x7f834404c690 tx=0x7f834400f220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:05.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 secure :-1 s=READY pgs=5118 cs=0 l=1 rev1=1 rx=0x7f834404c690 tx=0x7f834400f220).stop 2021-03-09T09:51:05.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:05.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 unknown :-1 s=CLOSED pgs=5118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:05.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:05.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:05.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:05.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019a10 con 0x7f8324003400 2021-03-09T09:51:05.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f833801cbc0 2021-03-09T09:51:05.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8324005e00 2021-03-09T09:51:05.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:51:05.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:05.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:51:05.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:05.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:05.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:05.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:05.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:05.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:05.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064e40 con 0x7f833801cbc0 2021-03-09T09:51:05.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5127 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f834400f220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:05.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833801cbc0 2021-03-09T09:51:05.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f833801cbc0 2021-03-09T09:51:05.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:05.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344032140 con 0x7f833801cbc0 2021-03-09T09:51:05.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:51:05 smithi146 bash[13136]: debug 2021-03-09T09:51:05.379+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14604 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:51:10.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:51:10.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:10.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5127 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f834400f220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:10.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5127 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f834400f220).stop 2021-03-09T09:51:10.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:10.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:10.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:10.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:51:10.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:10.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:10.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:10.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8324003400 2021-03-09T09:51:10.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ed70 con 0x7f833802d7b0 2021-03-09T09:51:10.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019a10 con 0x7f8324005e00 2021-03-09T09:51:10.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:51:10.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:10.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:10.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:10.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:10.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:10.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:10.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.384+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068480 con 0x7f833802d7b0 2021-03-09T09:51:10.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019bf0 secure :-1 s=READY pgs=5136 cs=0 l=1 rev1=1 rx=0x7f833c0624f0 tx=0x7f833c01c5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:10.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f833802d7b0 2021-03-09T09:51:10.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c054ed0 con 0x7f833802d7b0 2021-03-09T09:51:10.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:10.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833802d7b0 2021-03-09T09:51:10.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:51:10 smithi146 bash[13136]: debug 2021-03-09T09:51:10.383+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14629 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:51:15.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338019bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:51:15.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338019bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:15.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019bf0 secure :-1 s=READY pgs=5136 cs=0 l=1 rev1=1 rx=0x7f833c0624f0 tx=0x7f833c01c5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:15.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019bf0 secure :-1 s=READY pgs=5136 cs=0 l=1 rev1=1 rx=0x7f833c0624f0 tx=0x7f833c01c5d0).stop 2021-03-09T09:51:15.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338019bf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:15.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338019bf0 unknown :-1 s=CLOSED pgs=5136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:15.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:15.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:15.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:15.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:51:15.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e0a0 con 0x7f833801cbc0 2021-03-09T09:51:15.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ed70 con 0x7f8324005e00 2021-03-09T09:51:15.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.384+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:51:15.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.384+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:15.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.384+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:51:15.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:15.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:15.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.384+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:15.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:15.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.384+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:15.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:15.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.384+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380214d0 con 0x7f833801cbc0 2021-03-09T09:51:15.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5145 cs=0 l=1 rev1=1 rx=0x7f833c091490 tx=0x7f833c01c5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:15.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018490 con 0x7f833801cbc0 2021-03-09T09:51:15.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c054ed0 con 0x7f833801cbc0 2021-03-09T09:51:15.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:15.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f833801cbc0 2021-03-09T09:51:15.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:51:15 smithi146 bash[13136]: debug 2021-03-09T09:51:15.387+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14655 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:51:20.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.384+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:51:20.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.384+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:20.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5145 cs=0 l=1 rev1=1 rx=0x7f833c091490 tx=0x7f833c01c5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:20.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 secure :-1 s=READY pgs=5145 cs=0 l=1 rev1=1 rx=0x7f833c091490 tx=0x7f833c01c5d0).stop 2021-03-09T09:51:20.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:20.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.384+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833801cbc0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:20.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.384+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:20.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.384+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:20.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.384+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:20.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a760 con 0x7f8324003400 2021-03-09T09:51:20.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d670 con 0x7f833802d7b0 2021-03-09T09:51:20.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:51:20.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.384+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:51:20.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.384+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:20.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.384+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:51:20.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:20.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:20.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.388+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:20.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:20.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.388+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:20.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:20.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.388+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007dc0 con 0x7f833802d7b0 2021-03-09T09:51:20.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 secure :-1 s=READY pgs=5155 cs=0 l=1 rev1=1 rx=0x7f833c01c640 tx=0x7f833c01c5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:20.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c029e10 con 0x7f833802d7b0 2021-03-09T09:51:20.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c061ce0 con 0x7f833802d7b0 2021-03-09T09:51:20.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:20.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05b480 con 0x7f833802d7b0 2021-03-09T09:51:20.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:51:20 smithi146 bash[13136]: debug 2021-03-09T09:51:20.387+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14676 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:51:25.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:51:25.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:25.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 secure :-1 s=READY pgs=5155 cs=0 l=1 rev1=1 rx=0x7f833c01c640 tx=0x7f833c01c5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:25.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 secure :-1 s=READY pgs=5155 cs=0 l=1 rev1=1 rx=0x7f833c01c640 tx=0x7f833c01c5d0).stop 2021-03-09T09:51:25.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:25.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 unknown :-1 s=CLOSED pgs=5155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:25.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:25.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:25.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:25.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024340 con 0x7f8324003400 2021-03-09T09:51:25.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ed70 con 0x7f83380091b0 2021-03-09T09:51:25.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:51:25.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:51:25.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:51:25.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:25.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:25.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:25.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:25.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:25.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:25.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:25.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380068e0 con 0x7f83380091b0 2021-03-09T09:51:25.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5165 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c01c5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:25.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05cc30 con 0x7f83380091b0 2021-03-09T09:51:25.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05cc30 con 0x7f83380091b0 2021-03-09T09:51:25.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:25.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c045000 con 0x7f83380091b0 2021-03-09T09:51:25.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:51:25 smithi146 bash[13136]: debug 2021-03-09T09:51:25.391+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14695 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:51:30.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:51:30.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:30.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5165 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c01c5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:30.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5165 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c01c5d0).stop 2021-03-09T09:51:30.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:30.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:30.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:30.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:30.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:30.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018f90 con 0x7f8324003400 2021-03-09T09:51:30.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f833802d7b0 2021-03-09T09:51:30.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064010 con 0x7f8324005e00 2021-03-09T09:51:30.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:51:30.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:51:30.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:30.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:30.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:30.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:30.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:30.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:30.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:30.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068480 con 0x7f833802d7b0 2021-03-09T09:51:30.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 secure :-1 s=READY pgs=5170 cs=0 l=1 rev1=1 rx=0x7f833c058010 tx=0x7f833c01c5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:30.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0914b0 con 0x7f833802d7b0 2021-03-09T09:51:30.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0912d0 con 0x7f833802d7b0 2021-03-09T09:51:30.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:30.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05b4f0 con 0x7f833802d7b0 2021-03-09T09:51:30.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:51:30 smithi146 bash[13136]: debug 2021-03-09T09:51:30.395+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14729 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:51:35.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:51:35.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:35.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 secure :-1 s=READY pgs=5170 cs=0 l=1 rev1=1 rx=0x7f833c058010 tx=0x7f833c01c5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:35.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 secure :-1 s=READY pgs=5170 cs=0 l=1 rev1=1 rx=0x7f833c058010 tx=0x7f833c01c5d0).stop 2021-03-09T09:51:35.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f8338071d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:35.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f8338071d30 unknown :-1 s=CLOSED pgs=5170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:35.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:35.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:35.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:35.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:51:35.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026720 con 0x7f83380091b0 2021-03-09T09:51:35.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016820 con 0x7f8324005e00 2021-03-09T09:51:35.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:51:35.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:35.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:35.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:51:35.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:35.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.396+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:35.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.396+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:35.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.396+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:35.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.396+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:35.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.396+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800b1e0 con 0x7f83380091b0 2021-03-09T09:51:35.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.396+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5179 cs=0 l=1 rev1=1 rx=0x7f834400d000 tx=0x7f834400f220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:35.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f83380091b0 2021-03-09T09:51:35.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440171c0 con 0x7f83380091b0 2021-03-09T09:51:35.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:35.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f83380091b0 2021-03-09T09:51:35.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:51:35 smithi146 bash[13136]: debug 2021-03-09T09:51:35.395+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14746 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:51:40.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:51:40.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:40.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5179 cs=0 l=1 rev1=1 rx=0x7f834400d000 tx=0x7f834400f220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:40.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5179 cs=0 l=1 rev1=1 rx=0x7f834400d000 tx=0x7f834400f220).stop 2021-03-09T09:51:40.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:40.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:40.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:40.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:40.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380686a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:40.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:51:40.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380157e0 con 0x7f833802d7b0 2021-03-09T09:51:40.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d670 con 0x7f8324005e00 2021-03-09T09:51:40.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:51:40.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:40.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:51:40.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:40.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.396+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380686a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:40.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:40.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:40.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:40.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:40.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062960 con 0x7f833802d7b0 2021-03-09T09:51:40.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380686a0 secure :-1 s=READY pgs=5188 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c01c5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:40.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c003680 con 0x7f833802d7b0 2021-03-09T09:51:40.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f833802d7b0 2021-03-09T09:51:40.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:40.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833802d7b0 2021-03-09T09:51:40.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:51:40 smithi146 bash[13136]: debug 2021-03-09T09:51:40.399+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14783 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:51:45.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.395+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380686a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:51:45.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.395+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380686a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:45.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.395+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380686a0 secure :-1 s=READY pgs=5188 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c01c5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:45.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.395+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380686a0 secure :-1 s=READY pgs=5188 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c01c5d0).stop 2021-03-09T09:51:45.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.399+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 msgr2=0x7f83380686a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:45.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.399+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802d7b0 0x7f83380686a0 unknown :-1 s=CLOSED pgs=5188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:45.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.399+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:45.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.399+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:45.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.399+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:45.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.399+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e620 con 0x7f8324003400 2021-03-09T09:51:45.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.399+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e0a0 con 0x7f83380091b0 2021-03-09T09:51:45.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.399+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019a10 con 0x7f8324005e00 2021-03-09T09:51:45.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.399+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:51:45.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.399+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:45.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.399+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:51:45.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.399+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:45.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.399+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:45.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.403+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:45.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.403+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:45.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.403+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:45.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.403+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:45.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.403+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380242f0 con 0x7f83380091b0 2021-03-09T09:51:45.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.403+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5200 cs=0 l=1 rev1=1 rx=0x7f833c007360 tx=0x7f833c01c5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:45.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.403+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f83380091b0 2021-03-09T09:51:45.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.403+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c060c30 con 0x7f83380091b0 2021-03-09T09:51:45.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:45.407+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01a270 con 0x7f83380091b0 2021-03-09T09:51:45.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:51:45 smithi146 bash[13136]: debug 2021-03-09T09:51:45.403+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14807 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:51:50.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.403+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:51:50.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.403+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:50.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.403+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5200 cs=0 l=1 rev1=1 rx=0x7f833c007360 tx=0x7f833c01c5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:50.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.403+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5200 cs=0 l=1 rev1=1 rx=0x7f833c007360 tx=0x7f833c01c5d0).stop 2021-03-09T09:51:50.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.403+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:50.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.403+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:50.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.403+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:50.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.403+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:50.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.403+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338072190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:50.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.403+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c340 con 0x7f8324003400 2021-03-09T09:51:50.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.403+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014f80 con 0x7f8338071d10 2021-03-09T09:51:50.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.403+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:51:50.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.403+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:51:50.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.403+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:50.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.403+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:51:50.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.403+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:50.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.407+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338072190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:50.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.407+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:50.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.407+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:50.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.407+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:50.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.407+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:50.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.407+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f8338071d10 2021-03-09T09:51:50.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.407+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338072190 secure :-1 s=READY pgs=5206 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c01c5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:50.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c060c30 con 0x7f8338071d10 2021-03-09T09:51:50.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c071240 con 0x7f8338071d10 2021-03-09T09:51:50.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:50.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c042c60 con 0x7f8338071d10 2021-03-09T09:51:50.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:51:50 smithi146 bash[13136]: debug 2021-03-09T09:51:50.403+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14834 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:51:55.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338072190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:51:55.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338072190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:55.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338072190 secure :-1 s=READY pgs=5206 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c01c5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:55.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338072190 secure :-1 s=READY pgs=5206 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c01c5d0).stop 2021-03-09T09:51:55.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338072190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:55.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338072190 unknown :-1 s=CLOSED pgs=5206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:55.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:55.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:55.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:55.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cf60 con 0x7f8324003400 2021-03-09T09:51:55.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e620 con 0x7f83380091b0 2021-03-09T09:51:55.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024bb0 con 0x7f8324005e00 2021-03-09T09:51:55.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:51:55.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:55.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:51:55.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:55.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.407+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:55.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:55.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:55.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:55.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:55.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338016250 con 0x7f83380091b0 2021-03-09T09:51:55.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5217 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:55.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c02e560 con 0x7f83380091b0 2021-03-09T09:51:55.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c054960 con 0x7f83380091b0 2021-03-09T09:51:55.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:51:55.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02e350 con 0x7f83380091b0 2021-03-09T09:51:55.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:51:55 smithi146 bash[13136]: debug 2021-03-09T09:51:55.407+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14856 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:52:00.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.407+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:52:00.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.407+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:00.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.407+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5217 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:00.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.407+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5217 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c04de90).stop 2021-03-09T09:52:00.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.407+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:00.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.407+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:00.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.407+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338065fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:00.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.407+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:00.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.407+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:00.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.407+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e0a0 con 0x7f8324003400 2021-03-09T09:52:00.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.407+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8338071d10 2021-03-09T09:52:00.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.407+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380186d0 con 0x7f8324005e00 2021-03-09T09:52:00.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.411+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:52:00.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:52:00.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.411+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:00.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:00.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338065fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:00.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:00.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:00.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:00.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:00.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f8338071d10 2021-03-09T09:52:00.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338065fe0 secure :-1 s=READY pgs=5222 cs=0 l=1 rev1=1 rx=0x7f833c00fd40 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:00.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c039480 con 0x7f8338071d10 2021-03-09T09:52:00.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c039250 con 0x7f8338071d10 2021-03-09T09:52:00.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:00.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f8338071d10 2021-03-09T09:52:00.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:52:00 smithi146 bash[13136]: debug 2021-03-09T09:52:00.410+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14877 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:52:05.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338065fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:52:05.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338065fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:05.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338065fe0 secure :-1 s=READY pgs=5222 cs=0 l=1 rev1=1 rx=0x7f833c00fd40 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:05.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338065fe0 secure :-1 s=READY pgs=5222 cs=0 l=1 rev1=1 rx=0x7f833c00fd40 tx=0x7f833c04de90).stop 2021-03-09T09:52:05.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338065fe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:05.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338065fe0 unknown :-1 s=CLOSED pgs=5222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:05.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:05.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:05.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:05.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071650 con 0x7f8324003400 2021-03-09T09:52:05.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014f80 con 0x7f83380091b0 2021-03-09T09:52:05.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e0a0 con 0x7f8324005e00 2021-03-09T09:52:05.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:52:05.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:05.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:52:05.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:05.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:05.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:05.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:05.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:05.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:05.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028cf0 con 0x7f83380091b0 2021-03-09T09:52:05.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5232 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c01c430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:05.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c053450 con 0x7f83380091b0 2021-03-09T09:52:05.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c053270 con 0x7f83380091b0 2021-03-09T09:52:05.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:05.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02ecb0 con 0x7f83380091b0 2021-03-09T09:52:05.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:52:05 smithi146 bash[13136]: debug 2021-03-09T09:52:05.414+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14909 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:52:10.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:52:10.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:10.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5232 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c01c430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:10.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5232 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c01c430).stop 2021-03-09T09:52:10.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:10.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:10.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:10.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:10.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:10.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380164e0 con 0x7f8324003400 2021-03-09T09:52:10.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007e10 con 0x7f8338071d10 2021-03-09T09:52:10.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8324005e00 2021-03-09T09:52:10.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:52:10.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:10.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:52:10.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:10.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:10.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.415+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.419+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e610 con 0x7f8338071d10 2021-03-09T09:52:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.419+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 secure :-1 s=READY pgs=5240 cs=0 l=1 rev1=1 rx=0x7f8344054ec0 tx=0x7f834400f220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:10.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T09:52:10.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T09:52:10.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:10.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834401ebc0 con 0x7f8338071d10 2021-03-09T09:52:10.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:52:10 smithi146 bash[13136]: debug 2021-03-09T09:52:10.414+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14927 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:52:15.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:52:15.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:15.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 secure :-1 s=READY pgs=5240 cs=0 l=1 rev1=1 rx=0x7f8344054ec0 tx=0x7f834400f220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:15.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 secure :-1 s=READY pgs=5240 cs=0 l=1 rev1=1 rx=0x7f8344054ec0 tx=0x7f834400f220).stop 2021-03-09T09:52:15.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801ab20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:15.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 unknown :-1 s=CLOSED pgs=5240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:15.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:15.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:15.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:15.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:52:15.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338029710 con 0x7f8324003400 2021-03-09T09:52:15.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:15.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:52:15.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:15.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f83380091b0 2021-03-09T09:52:15.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e0a0 con 0x7f8324005e00 2021-03-09T09:52:15.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:15.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:15.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:15.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:15.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.419+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:15.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.423+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833805eaa0 con 0x7f83380091b0 2021-03-09T09:52:15.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.423+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5249 cs=0 l=1 rev1=1 rx=0x7f8344079d70 tx=0x7f834400f220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:15.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344009040 con 0x7f83380091b0 2021-03-09T09:52:15.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.427+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f83380091b0 2021-03-09T09:52:15.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:15.427+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405ea60 con 0x7f83380091b0 2021-03-09T09:52:15.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:52:15 smithi146 bash[13136]: debug 2021-03-09T09:52:15.418+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14959 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:52:20.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.419+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:52:20.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.419+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:20.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.419+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5249 cs=0 l=1 rev1=1 rx=0x7f8344079d70 tx=0x7f834400f220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:20.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.419+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5249 cs=0 l=1 rev1=1 rx=0x7f8344079d70 tx=0x7f834400f220).stop 2021-03-09T09:52:20.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:20.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.419+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:20.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.419+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338017720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:20.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.419+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:20.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.419+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:20.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ef70 con 0x7f8324003400 2021-03-09T09:52:20.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014690 con 0x7f8338071d10 2021-03-09T09:52:20.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324005e00 2021-03-09T09:52:20.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.419+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:52:20.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.419+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:20.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.423+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:52:20.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.423+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:20.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.423+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338017720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:20.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.423+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:20.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.423+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:20.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.423+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:20.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.423+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:20.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.423+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e0a0 con 0x7f8338071d10 2021-03-09T09:52:20.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.423+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338017720 secure :-1 s=READY pgs=5258 cs=0 l=1 rev1=1 rx=0x7f833c007880 tx=0x7f833c01c430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:20.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.427+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c053270 con 0x7f8338071d10 2021-03-09T09:52:20.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.427+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05ae20 con 0x7f8338071d10 2021-03-09T09:52:20.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:20.431+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05aab0 con 0x7f8338071d10 2021-03-09T09:52:20.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:52:20 smithi146 bash[13136]: debug 2021-03-09T09:52:20.422+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 14986 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:52:25.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338017720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:52:25.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338017720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:25.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338017720 secure :-1 s=READY pgs=5258 cs=0 l=1 rev1=1 rx=0x7f833c007880 tx=0x7f833c01c430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:25.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338017720 secure :-1 s=READY pgs=5258 cs=0 l=1 rev1=1 rx=0x7f833c007880 tx=0x7f833c01c430).stop 2021-03-09T09:52:25.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338017720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:25.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338017720 unknown :-1 s=CLOSED pgs=5258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:25.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:25.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:25.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:25.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380215e0 con 0x7f8324003400 2021-03-09T09:52:25.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f83380091b0 2021-03-09T09:52:25.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020680 con 0x7f8324005e00 2021-03-09T09:52:25.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:52:25.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:25.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:52:25.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:25.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.423+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:25.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.427+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:25.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.427+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:25.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.427+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:25.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.427+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:25.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.427+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007730 con 0x7f83380091b0 2021-03-09T09:52:25.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.427+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5266 cs=0 l=1 rev1=1 rx=0x7f834402b3e0 tx=0x7f834400f220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:25.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.431+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401ec60 con 0x7f83380091b0 2021-03-09T09:52:25.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.431+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f83380091b0 2021-03-09T09:52:25.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:25.435+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440171c0 con 0x7f83380091b0 2021-03-09T09:52:25.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:52:25 smithi146 bash[13136]: debug 2021-03-09T09:52:25.426+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15009 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:52:30.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:52:30.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:30.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5266 cs=0 l=1 rev1=1 rx=0x7f834402b3e0 tx=0x7f834400f220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:30.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5266 cs=0 l=1 rev1=1 rx=0x7f834402b3e0 tx=0x7f834400f220).stop 2021-03-09T09:52:30.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:30.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:30.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:30.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:30.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380144c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:30.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014f80 con 0x7f8324003400 2021-03-09T09:52:30.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8338071d10 2021-03-09T09:52:30.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e2a0 con 0x7f8324005e00 2021-03-09T09:52:30.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:52:30.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:30.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:52:30.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:30.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.427+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380144c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:30.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.431+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:30.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.431+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:30.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.431+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:30.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.431+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:30.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.431+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e0a0 con 0x7f8338071d10 2021-03-09T09:52:30.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.431+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380144c0 secure :-1 s=READY pgs=5275 cs=0 l=1 rev1=1 rx=0x7f833c0454e0 tx=0x7f833c01c430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:30.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.431+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c003680 con 0x7f8338071d10 2021-03-09T09:52:30.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.431+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05ae20 con 0x7f8338071d10 2021-03-09T09:52:30.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:30.435+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f8338071d10 2021-03-09T09:52:30.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:52:30 smithi146 bash[13136]: debug 2021-03-09T09:52:30.426+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15031 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:52:35.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f83380144c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:52:35.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f83380144c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:35.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380144c0 secure :-1 s=READY pgs=5275 cs=0 l=1 rev1=1 rx=0x7f833c0454e0 tx=0x7f833c01c430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:35.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380144c0 secure :-1 s=READY pgs=5275 cs=0 l=1 rev1=1 rx=0x7f833c0454e0 tx=0x7f833c01c430).stop 2021-03-09T09:52:35.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f83380144c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:35.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380144c0 unknown :-1 s=CLOSED pgs=5275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:35.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:35.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:35.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:35.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324003400 2021-03-09T09:52:35.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006860 con 0x7f83380091b0 2021-03-09T09:52:35.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071b80 con 0x7f8324005e00 2021-03-09T09:52:35.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:52:35.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:52:35.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:35.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:35.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.430+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:35.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.434+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:35.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.434+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:35.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.434+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:35.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.434+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:35.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.434+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338029710 con 0x7f83380091b0 2021-03-09T09:52:35.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.434+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5285 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c01c430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:35.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.434+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018620 con 0x7f83380091b0 2021-03-09T09:52:35.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.438+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c036b30 con 0x7f83380091b0 2021-03-09T09:52:35.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:35.438+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c026380 con 0x7f83380091b0 2021-03-09T09:52:35.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:52:35 smithi146 bash[13136]: debug 2021-03-09T09:52:35.430+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15063 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:52:40.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:52:40.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:40.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5285 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c01c430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:40.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5285 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c01c430).stop 2021-03-09T09:52:40.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:40.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:40.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:40.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:40.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:40.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014f80 con 0x7f8324003400 2021-03-09T09:52:40.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071b80 con 0x7f8338071d10 2021-03-09T09:52:40.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:52:40.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:52:40.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:40.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:40.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:52:40.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:40.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:40.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:40.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:40.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:40.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.434+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007730 con 0x7f8338071d10 2021-03-09T09:52:40.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.438+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 secure :-1 s=READY pgs=5293 cs=0 l=1 rev1=1 rx=0x7f833c0618d0 tx=0x7f833c01c430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:40.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.438+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c010350 con 0x7f8338071d10 2021-03-09T09:52:40.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.438+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c061710 con 0x7f8338071d10 2021-03-09T09:52:40.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:40.442+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c061bd0 con 0x7f8338071d10 2021-03-09T09:52:40.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:52:40 smithi146 bash[13136]: debug 2021-03-09T09:52:40.434+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15088 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:52:45.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:52:45.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:45.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 secure :-1 s=READY pgs=5293 cs=0 l=1 rev1=1 rx=0x7f833c0618d0 tx=0x7f833c01c430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:45.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 secure :-1 s=READY pgs=5293 cs=0 l=1 rev1=1 rx=0x7f833c0618d0 tx=0x7f833c01c430).stop 2021-03-09T09:52:45.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801ab20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:45.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 unknown :-1 s=CLOSED pgs=5293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:45.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:45.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338018040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:45.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:45.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:52:45.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fbd0 con 0x7f83380091b0 2021-03-09T09:52:45.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d330 con 0x7f8324005e00 2021-03-09T09:52:45.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:52:45.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:45.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:52:45.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:45.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.434+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338018040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:45.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.438+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:45.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.438+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:45.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.438+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:45.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.438+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:45.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.438+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380149e0 con 0x7f83380091b0 2021-03-09T09:52:45.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.438+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338018040 secure :-1 s=READY pgs=5300 cs=0 l=1 rev1=1 rx=0x7f83440162e0 tx=0x7f834400f220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:45.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.442+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344072580 con 0x7f83380091b0 2021-03-09T09:52:45.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.442+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033030 con 0x7f83380091b0 2021-03-09T09:52:45.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:45.446+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440171c0 con 0x7f83380091b0 2021-03-09T09:52:45.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:52:45 smithi146 bash[13136]: debug 2021-03-09T09:52:45.438+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15114 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:52:50.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338018040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:52:50.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338018040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:50.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338018040 secure :-1 s=READY pgs=5300 cs=0 l=1 rev1=1 rx=0x7f83440162e0 tx=0x7f834400f220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:50.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338018040 secure :-1 s=READY pgs=5300 cs=0 l=1 rev1=1 rx=0x7f83440162e0 tx=0x7f834400f220).stop 2021-03-09T09:52:50.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338018040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:50.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338018040 unknown :-1 s=CLOSED pgs=5300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:50.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:50.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:50.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:50.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064e50 con 0x7f8324003400 2021-03-09T09:52:50.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071b80 con 0x7f8338071d10 2021-03-09T09:52:50.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014f80 con 0x7f8324005e00 2021-03-09T09:52:50.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:52:50.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:50.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:52:50.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:50.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.438+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:50.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.442+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:50.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.442+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:50.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.442+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:50.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.442+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:50.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.442+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007730 con 0x7f8338071d10 2021-03-09T09:52:50.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.442+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 secure :-1 s=READY pgs=5309 cs=0 l=1 rev1=1 rx=0x7f833c030040 tx=0x7f833c01c430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:50.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.442+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f8338071d10 2021-03-09T09:52:50.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.442+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0094a0 con 0x7f8338071d10 2021-03-09T09:52:50.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:50.446+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0412e0 con 0x7f8338071d10 2021-03-09T09:52:50.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:52:50 smithi146 bash[13136]: debug 2021-03-09T09:52:50.438+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15139 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:52:55.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:52:55.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:55.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 secure :-1 s=READY pgs=5309 cs=0 l=1 rev1=1 rx=0x7f833c030040 tx=0x7f833c01c430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:55.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 secure :-1 s=READY pgs=5309 cs=0 l=1 rev1=1 rx=0x7f833c030040 tx=0x7f833c01c430).stop 2021-03-09T09:52:55.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801ab20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:55.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 unknown :-1 s=CLOSED pgs=5309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:55.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:55.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:55.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:52:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026720 con 0x7f83380091b0 2021-03-09T09:52:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801da80 con 0x7f8324005e00 2021-03-09T09:52:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:52:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:52:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.442+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338019120 con 0x7f83380091b0 2021-03-09T09:52:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.446+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5320 cs=0 l=1 rev1=1 rx=0x7f833c012c60 tx=0x7f833c01c430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.446+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f83380091b0 2021-03-09T09:52:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.446+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f83380091b0 2021-03-09T09:52:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:52:55.450+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f83380091b0 2021-03-09T09:52:55.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:52:55 smithi146 bash[13136]: debug 2021-03-09T09:52:55.442+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15162 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:53:00.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:53:00.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:00.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5320 cs=0 l=1 rev1=1 rx=0x7f833c012c60 tx=0x7f833c01c430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:00.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5320 cs=0 l=1 rev1=1 rx=0x7f833c012c60 tx=0x7f833c01c430).stop 2021-03-09T09:53:00.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:00.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:00.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:00.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:00.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:00.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063990 con 0x7f8324003400 2021-03-09T09:53:00.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071890 con 0x7f8338071d10 2021-03-09T09:53:00.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071b80 con 0x7f8324005e00 2021-03-09T09:53:00.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:53:00.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:00.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:53:00.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:00.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:00.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:00.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:00.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:00.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:00.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802cb40 con 0x7f8338071d10 2021-03-09T09:53:00.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 secure :-1 s=READY pgs=5326 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c01c430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:00.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f8338071d10 2021-03-09T09:53:00.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.446+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0296d0 con 0x7f8338071d10 2021-03-09T09:53:00.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:00.450+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0613a0 con 0x7f8338071d10 2021-03-09T09:53:00.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:53:00 smithi146 bash[13136]: debug 2021-03-09T09:53:00.446+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15175 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:53:05.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:53:05.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:05.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 secure :-1 s=READY pgs=5326 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c01c430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:05.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 secure :-1 s=READY pgs=5326 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c01c430).stop 2021-03-09T09:53:05.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801ab20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:05.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801ab20 unknown :-1 s=CLOSED pgs=5326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:05.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:05.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:05.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:53:05.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:05.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014f80 con 0x7f8324003400 2021-03-09T09:53:05.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f83380091b0 2021-03-09T09:53:05.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T09:53:05.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:05.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:05.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:53:05.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.446+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:05.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.450+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:05.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.450+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:05.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.450+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:05.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.450+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:05.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.450+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338019120 con 0x7f83380091b0 2021-03-09T09:53:05.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.450+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5335 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834400f220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:05.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.450+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401f120 con 0x7f83380091b0 2021-03-09T09:53:05.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.450+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f83380091b0 2021-03-09T09:53:05.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:05.454+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f83380091b0 2021-03-09T09:53:05.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:53:05 smithi146 bash[13136]: debug 2021-03-09T09:53:05.449+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15211 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:53:10.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:53:10.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5335 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834400f220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5335 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834400f220).stop 2021-03-09T09:53:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014e40 con 0x7f8324003400 2021-03-09T09:53:10.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071890 con 0x7f8338071d10 2021-03-09T09:53:10.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071b80 con 0x7f8324005e00 2021-03-09T09:53:10.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:53:10.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:53:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.450+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.454+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.454+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.454+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.454+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026af0 con 0x7f8338071d10 2021-03-09T09:53:10.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.454+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380100b0 secure :-1 s=READY pgs=5344 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01c430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:10.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.454+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0626c0 con 0x7f8338071d10 2021-03-09T09:53:10.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.454+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0077b0 con 0x7f8338071d10 2021-03-09T09:53:10.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:10.458+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338071d10 2021-03-09T09:53:10.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:53:10 smithi146 bash[13136]: debug 2021-03-09T09:53:10.449+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15237 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:53:15.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:53:15.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:15.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380100b0 secure :-1 s=READY pgs=5344 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01c430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:15.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380100b0 secure :-1 s=READY pgs=5344 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01c430).stop 2021-03-09T09:53:15.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f83380100b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:15.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380100b0 unknown :-1 s=CLOSED pgs=5344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:15.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:15.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:15.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:15.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012810 con 0x7f8324003400 2021-03-09T09:53:15.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f83380091b0 2021-03-09T09:53:15.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324005e00 2021-03-09T09:53:15.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:53:15.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:15.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:15.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:53:15.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:15.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:15.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:15.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:15.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:15.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338019120 con 0x7f83380091b0 2021-03-09T09:53:15.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.454+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5352 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834400f220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:15.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.458+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402ba10 con 0x7f83380091b0 2021-03-09T09:53:15.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.458+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344010250 con 0x7f83380091b0 2021-03-09T09:53:15.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:15.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f83380091b0 2021-03-09T09:53:15.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:53:15 smithi146 bash[13136]: debug 2021-03-09T09:53:15.453+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15260 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:53:20.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.454+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:53:20.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.454+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:20.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.454+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5352 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834400f220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:20.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.454+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5352 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834400f220).stop 2021-03-09T09:53:20.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.454+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:20.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.454+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:20.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.454+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:20.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.454+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:20.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.454+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:20.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.454+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015a00 con 0x7f8324003400 2021-03-09T09:53:20.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.454+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d2f0 con 0x7f8338071d10 2021-03-09T09:53:20.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.454+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019a10 con 0x7f8324005e00 2021-03-09T09:53:20.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.454+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:53:20.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.454+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:20.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.458+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:53:20.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.458+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:20.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.458+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:20.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.458+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:20.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.458+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:20.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.458+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:20.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.458+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:20.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.458+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007730 con 0x7f8338071d10 2021-03-09T09:53:20.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.458+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380100b0 secure :-1 s=READY pgs=5361 cs=0 l=1 rev1=1 rx=0x7f833c061d20 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:20.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.458+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018620 con 0x7f8338071d10 2021-03-09T09:53:20.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.458+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c018620 con 0x7f8338071d10 2021-03-09T09:53:20.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:20.462+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338071d10 2021-03-09T09:53:20.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:53:20 smithi146 bash[13136]: debug 2021-03-09T09:53:20.457+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15288 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:53:25.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:53:25.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:25.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380100b0 secure :-1 s=READY pgs=5361 cs=0 l=1 rev1=1 rx=0x7f833c061d20 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:25.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380100b0 secure :-1 s=READY pgs=5361 cs=0 l=1 rev1=1 rx=0x7f833c061d20 tx=0x7f833c04de90).stop 2021-03-09T09:53:25.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f83380100b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:25.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380100b0 unknown :-1 s=CLOSED pgs=5361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:25.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:25.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833801b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:25.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:25.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T09:53:25.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071b80 con 0x7f83380091b0 2021-03-09T09:53:25.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012810 con 0x7f8324005e00 2021-03-09T09:53:25.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:53:25.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833801b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:25.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:25.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:53:25.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.457+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:25.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.461+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:25.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.461+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:25.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.461+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:25.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.461+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:25.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.461+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a430 con 0x7f83380091b0 2021-03-09T09:53:25.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.461+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833801b240 secure :-1 s=READY pgs=5370 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:25.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.461+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400fec0 con 0x7f83380091b0 2021-03-09T09:53:25.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.461+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344017e50 con 0x7f83380091b0 2021-03-09T09:53:25.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:25.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f83380091b0 2021-03-09T09:53:25.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:53:25 smithi146 bash[13136]: debug 2021-03-09T09:53:25.457+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15313 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:53:30.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833801b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:53:30.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833801b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:30.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833801b240 secure :-1 s=READY pgs=5370 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:30.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833801b240 secure :-1 s=READY pgs=5370 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834404c330).stop 2021-03-09T09:53:30.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833801b240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:30.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833801b240 unknown :-1 s=CLOSED pgs=5370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:30.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:30.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:30.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338015120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:30.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324003400 2021-03-09T09:53:30.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064000 con 0x7f8338071d10 2021-03-09T09:53:30.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801fda0 con 0x7f8324005e00 2021-03-09T09:53:30.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:53:30.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:30.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:53:30.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:30.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.461+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338015120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:30.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.465+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:30.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.465+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:30.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.465+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:30.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.465+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:30.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.465+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024d90 con 0x7f8338071d10 2021-03-09T09:53:30.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.465+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338015120 secure :-1 s=READY pgs=5381 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:30.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.465+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f8338071d10 2021-03-09T09:53:30.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f8338071d10 2021-03-09T09:53:30.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:30.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c061d50 con 0x7f8338071d10 2021-03-09T09:53:30.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:53:30 smithi146 bash[13136]: debug 2021-03-09T09:53:30.461+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15337 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:53:35.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.461+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338015120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:53:35.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.461+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338015120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:35.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.461+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338015120 secure :-1 s=READY pgs=5381 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:35.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.461+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338015120 secure :-1 s=READY pgs=5381 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c04de90).stop 2021-03-09T09:53:35.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.461+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338015120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:35.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.461+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338015120 unknown :-1 s=CLOSED pgs=5381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:35.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:35.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338016030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:35.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:35.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:53:35.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071b80 con 0x7f83380091b0 2021-03-09T09:53:35.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064000 con 0x7f8324005e00 2021-03-09T09:53:35.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:53:35.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:35.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:53:35.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:35.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338016030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:35.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:35.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:35.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:35.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:35.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026af0 con 0x7f83380091b0 2021-03-09T09:53:35.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.465+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338016030 secure :-1 s=READY pgs=5387 cs=0 l=1 rev1=1 rx=0x7f8338020320 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:35.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404b6f0 con 0x7f83380091b0 2021-03-09T09:53:35.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404b920 con 0x7f83380091b0 2021-03-09T09:53:35.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:35.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f83380091b0 2021-03-09T09:53:35.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:53:35 smithi146 bash[13136]: debug 2021-03-09T09:53:35.465+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15361 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:53:40.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338016030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:53:40.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338016030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:40.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338016030 secure :-1 s=READY pgs=5387 cs=0 l=1 rev1=1 rx=0x7f8338020320 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:40.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338016030 secure :-1 s=READY pgs=5387 cs=0 l=1 rev1=1 rx=0x7f8338020320 tx=0x7f834404c330).stop 2021-03-09T09:53:40.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338016030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:40.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338016030 unknown :-1 s=CLOSED pgs=5387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:40.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:40.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:40.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:40.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021950 con 0x7f8324003400 2021-03-09T09:53:40.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8338071d10 2021-03-09T09:53:40.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324005e00 2021-03-09T09:53:40.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:53:40.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:40.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:53:40.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:40.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.469+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:40.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:40.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:40.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:40.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:40.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007700 con 0x7f8338071d10 2021-03-09T09:53:40.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 secure :-1 s=READY pgs=5397 cs=0 l=1 rev1=1 rx=0x7f833c040c90 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:40.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f8338071d10 2021-03-09T09:53:40.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338071d10 2021-03-09T09:53:40.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:40.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c062350 con 0x7f8338071d10 2021-03-09T09:53:40.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:53:40 smithi146 bash[13136]: debug 2021-03-09T09:53:40.469+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15387 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:53:45.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338028460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:53:45.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338028460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:45.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 secure :-1 s=READY pgs=5397 cs=0 l=1 rev1=1 rx=0x7f833c040c90 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:45.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 secure :-1 s=READY pgs=5397 cs=0 l=1 rev1=1 rx=0x7f833c040c90 tx=0x7f833c04de90).stop 2021-03-09T09:53:45.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338028460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:45.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 unknown :-1 s=CLOSED pgs=5397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:45.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:45.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:45.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:45.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026720 con 0x7f8324003400 2021-03-09T09:53:45.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019320 con 0x7f83380091b0 2021-03-09T09:53:45.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024f70 con 0x7f8324005e00 2021-03-09T09:53:45.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:53:45.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:45.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:45.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:53:45.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:45.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:45.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:45.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:45.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:45.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380117b0 con 0x7f83380091b0 2021-03-09T09:53:45.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5404 cs=0 l=1 rev1=1 rx=0x7f833c01c470 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:45.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05f840 con 0x7f83380091b0 2021-03-09T09:53:45.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f83380091b0 2021-03-09T09:53:45.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:45.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05f200 con 0x7f83380091b0 2021-03-09T09:53:45.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:53:45 smithi146 bash[13136]: debug 2021-03-09T09:53:45.469+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15415 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:53:50.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:53:50.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:50.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5404 cs=0 l=1 rev1=1 rx=0x7f833c01c470 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:50.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5404 cs=0 l=1 rev1=1 rx=0x7f833c01c470 tx=0x7f833c04de90).stop 2021-03-09T09:53:50.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:50.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:50.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:50.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:50.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:50.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071b80 con 0x7f8324003400 2021-03-09T09:53:50.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021950 con 0x7f8338071d10 2021-03-09T09:53:50.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802da10 con 0x7f8324005e00 2021-03-09T09:53:50.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:53:50.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:50.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:53:50.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:50.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.473+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:50.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:50.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:50.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:50.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:50.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380117b0 con 0x7f8338071d10 2021-03-09T09:53:50.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 secure :-1 s=READY pgs=5415 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:50.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401f910 con 0x7f8338071d10 2021-03-09T09:53:50.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T09:53:50.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:50.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344049710 con 0x7f8338071d10 2021-03-09T09:53:50.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:53:50 smithi146 bash[13136]: debug 2021-03-09T09:53:50.473+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15437 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:53:55.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.473+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338028460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:53:55.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.473+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338028460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:55.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.473+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 secure :-1 s=READY pgs=5415 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:55.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.473+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 secure :-1 s=READY pgs=5415 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834404c330).stop 2021-03-09T09:53:55.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338028460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:55.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 unknown :-1 s=CLOSED pgs=5415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:55.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:55.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:55.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.473+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:55.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324003400 2021-03-09T09:53:55.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f83380091b0 2021-03-09T09:53:55.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.473+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017920 con 0x7f8324005e00 2021-03-09T09:53:55.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.477+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:53:55.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.477+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:55.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.477+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:53:55.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.477+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:55.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:55.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:55.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:55.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:55.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:55.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a430 con 0x7f83380091b0 2021-03-09T09:53:55.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5422 cs=0 l=1 rev1=1 rx=0x7f834401fde0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:55.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440783a0 con 0x7f83380091b0 2021-03-09T09:53:55.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344032b20 con 0x7f83380091b0 2021-03-09T09:53:55.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:53:55.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834401f350 con 0x7f83380091b0 2021-03-09T09:53:55.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:53:55 smithi146 bash[13136]: debug 2021-03-09T09:53:55.477+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15462 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:54:00.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:54:00.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.477+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:00.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5422 cs=0 l=1 rev1=1 rx=0x7f834401fde0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:00.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.477+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5422 cs=0 l=1 rev1=1 rx=0x7f834401fde0 tx=0x7f834404c330).stop 2021-03-09T09:54:00.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:00.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:00.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:00.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:00.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.477+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:00.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024f70 con 0x7f8324003400 2021-03-09T09:54:00.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8338071d10 2021-03-09T09:54:00.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.477+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071b80 con 0x7f8324005e00 2021-03-09T09:54:00.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.477+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:54:00.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.477+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:00.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.477+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:54:00.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.477+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:00.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.481+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:00.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.481+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:00.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.481+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:00.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.481+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:00.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.481+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:00.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.481+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065400 con 0x7f8338071d10 2021-03-09T09:54:00.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.481+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 secure :-1 s=READY pgs=5430 cs=0 l=1 rev1=1 rx=0x7f83440499b0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:00.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440179e0 con 0x7f8338071d10 2021-03-09T09:54:00.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834401f8e0 con 0x7f8338071d10 2021-03-09T09:54:00.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:00.489+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344032b20 con 0x7f8338071d10 2021-03-09T09:54:00.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:54:00 smithi146 bash[13136]: debug 2021-03-09T09:54:00.481+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15492 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:54:05.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.481+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338028460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:54:05.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.481+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338028460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:05.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.481+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 secure :-1 s=READY pgs=5430 cs=0 l=1 rev1=1 rx=0x7f83440499b0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:05.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.481+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 secure :-1 s=READY pgs=5430 cs=0 l=1 rev1=1 rx=0x7f83440499b0 tx=0x7f834404c330).stop 2021-03-09T09:54:05.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338028460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:05.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.481+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 unknown :-1 s=CLOSED pgs=5430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:05.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.481+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:05.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.481+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:05.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.481+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:05.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380668c0 con 0x7f8324003400 2021-03-09T09:54:05.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026760 con 0x7f83380091b0 2021-03-09T09:54:05.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.481+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023c00 con 0x7f8324005e00 2021-03-09T09:54:05.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.481+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:54:05.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.481+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:05.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.481+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:54:05.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.481+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:05.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.485+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:05.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.485+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:05.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.485+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:05.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.485+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:05.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.485+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:05.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.485+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010c70 con 0x7f83380091b0 2021-03-09T09:54:05.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.485+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5440 cs=0 l=1 rev1=1 rx=0x7f833c01d970 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:05.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f83380091b0 2021-03-09T09:54:05.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c018620 con 0x7f83380091b0 2021-03-09T09:54:05.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:05.489+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f83380091b0 2021-03-09T09:54:05.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:54:05 smithi146 bash[13136]: debug 2021-03-09T09:54:05.480+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15516 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:54:10.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:54:10.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:10.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5440 cs=0 l=1 rev1=1 rx=0x7f833c01d970 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:10.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5440 cs=0 l=1 rev1=1 rx=0x7f833c01d970 tx=0x7f833c04de90).stop 2021-03-09T09:54:10.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:10.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:10.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:10.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:10.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:10.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324003400 2021-03-09T09:54:10.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022030 con 0x7f8338071d10 2021-03-09T09:54:10.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017920 con 0x7f8324005e00 2021-03-09T09:54:10.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:54:10.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:10.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:54:10.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:10.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:10.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:10.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:10.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:10.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:10.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.485+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380117b0 con 0x7f8338071d10 2021-03-09T09:54:10.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.489+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 secure :-1 s=READY pgs=5448 cs=0 l=1 rev1=1 rx=0x7f833c091870 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:10.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.489+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018620 con 0x7f8338071d10 2021-03-09T09:54:10.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.493+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0918e0 con 0x7f8338071d10 2021-03-09T09:54:10.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:10.497+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06baf0 con 0x7f8338071d10 2021-03-09T09:54:10.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:54:10 smithi146 bash[13136]: debug 2021-03-09T09:54:10.484+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15532 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:54:15.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338028460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:54:15.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338028460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:15.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 secure :-1 s=READY pgs=5448 cs=0 l=1 rev1=1 rx=0x7f833c091870 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:15.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 secure :-1 s=READY pgs=5448 cs=0 l=1 rev1=1 rx=0x7f833c091870 tx=0x7f833c04de90).stop 2021-03-09T09:54:15.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338028460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:15.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028460 unknown :-1 s=CLOSED pgs=5448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:15.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:15.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:15.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:15.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ffe0 con 0x7f8324003400 2021-03-09T09:54:15.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071b80 con 0x7f83380091b0 2021-03-09T09:54:15.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014050 con 0x7f8324005e00 2021-03-09T09:54:15.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:54:15.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:15.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:54:15.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:15.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:15.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:15.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:15.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:15.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:15.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800d4b0 con 0x7f83380091b0 2021-03-09T09:54:15.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5456 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:15.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c078e90 con 0x7f83380091b0 2021-03-09T09:54:15.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c007410 con 0x7f83380091b0 2021-03-09T09:54:15.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:15.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0412e0 con 0x7f83380091b0 2021-03-09T09:54:15.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:54:15 smithi146 bash[13136]: debug 2021-03-09T09:54:15.488+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15567 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:54:20.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:54:20.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:20.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5456 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:20.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5456 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04de90).stop 2021-03-09T09:54:20.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:20.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:20.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:20.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:20.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:20.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380668c0 con 0x7f8324003400 2021-03-09T09:54:20.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064e10 con 0x7f8338071d10 2021-03-09T09:54:20.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019320 con 0x7f8324005e00 2021-03-09T09:54:20.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:54:20.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:20.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:54:20.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:20.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:20.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:20.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:20.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:20.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:20.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801d2b0 con 0x7f8338071d10 2021-03-09T09:54:20.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.492+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338006e50 secure :-1 s=READY pgs=5465 cs=0 l=1 rev1=1 rx=0x7f834400e010 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:20.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344017770 con 0x7f8338071d10 2021-03-09T09:54:20.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344072190 con 0x7f8338071d10 2021-03-09T09:54:20.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:20.500+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440171c0 con 0x7f8338071d10 2021-03-09T09:54:20.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:54:20 smithi146 bash[13136]: debug 2021-03-09T09:54:20.492+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15594 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:54:25.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:54:25.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:25.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338006e50 secure :-1 s=READY pgs=5465 cs=0 l=1 rev1=1 rx=0x7f834400e010 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:25.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338006e50 secure :-1 s=READY pgs=5465 cs=0 l=1 rev1=1 rx=0x7f834400e010 tx=0x7f834404c330).stop 2021-03-09T09:54:25.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:25.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338006e50 unknown :-1 s=CLOSED pgs=5465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:25.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:25.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:25.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:25.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:54:25.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071b80 con 0x7f83380091b0 2021-03-09T09:54:25.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007e30 con 0x7f8324005e00 2021-03-09T09:54:25.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:54:25.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:25.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:54:25.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:25.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:25.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:25.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:25.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:25.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:25.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380149e0 con 0x7f83380091b0 2021-03-09T09:54:25.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.496+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5474 cs=0 l=1 rev1=1 rx=0x7f833c040c00 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:25.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.504+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0912d0 con 0x7f83380091b0 2021-03-09T09:54:25.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.504+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c018620 con 0x7f83380091b0 2021-03-09T09:54:25.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:25.504+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06bd00 con 0x7f83380091b0 2021-03-09T09:54:25.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:54:25 smithi146 bash[13136]: debug 2021-03-09T09:54:25.496+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15618 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:54:30.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:54:30.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:30.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5474 cs=0 l=1 rev1=1 rx=0x7f833c040c00 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:30.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5474 cs=0 l=1 rev1=1 rx=0x7f833c040c00 tx=0x7f833c04de90).stop 2021-03-09T09:54:30.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:30.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:30.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338007610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:30.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:30.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:30.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064e10 con 0x7f8324003400 2021-03-09T09:54:30.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8338071d10 2021-03-09T09:54:30.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010490 con 0x7f8324005e00 2021-03-09T09:54:30.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:54:30.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:30.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:54:30.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:30.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338007610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:30.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:30.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:30.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:30.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:30.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.496+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380117b0 con 0x7f8338071d10 2021-03-09T09:54:30.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.500+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338007610 secure :-1 s=READY pgs=5482 cs=0 l=1 rev1=1 rx=0x7f833c01afc0 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:30.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.500+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00f530 con 0x7f8338071d10 2021-03-09T09:54:30.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.500+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00f530 con 0x7f8338071d10 2021-03-09T09:54:30.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:30.504+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06bd00 con 0x7f8338071d10 2021-03-09T09:54:30.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:54:30 smithi146 bash[13136]: debug 2021-03-09T09:54:30.496+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15645 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:54:35.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.496+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338007610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:54:35.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.496+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338007610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:35.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.496+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338007610 secure :-1 s=READY pgs=5482 cs=0 l=1 rev1=1 rx=0x7f833c01afc0 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:35.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.496+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338007610 secure :-1 s=READY pgs=5482 cs=0 l=1 rev1=1 rx=0x7f833c01afc0 tx=0x7f833c04de90).stop 2021-03-09T09:54:35.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338007610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:35.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.496+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338007610 unknown :-1 s=CLOSED pgs=5482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:35.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:35.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:35.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:35.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380204d0 con 0x7f8324003400 2021-03-09T09:54:35.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019320 con 0x7f83380091b0 2021-03-09T09:54:35.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:54:35.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:54:35.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:35.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:54:35.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:35.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:35.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:35.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:35.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:35.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:35.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380117b0 con 0x7f83380091b0 2021-03-09T09:54:35.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.500+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5492 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:35.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.504+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00f300 con 0x7f83380091b0 2021-03-09T09:54:35.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.504+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c010940 con 0x7f83380091b0 2021-03-09T09:54:35.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:35.508+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0572b0 con 0x7f83380091b0 2021-03-09T09:54:35.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:54:35 smithi146 bash[13136]: debug 2021-03-09T09:54:35.500+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15666 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:54:40.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:54:40.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:40.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5492 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:40.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5492 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c04de90).stop 2021-03-09T09:54:40.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:40.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:40.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:40.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:40.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:40.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d160 con 0x7f8324003400 2021-03-09T09:54:40.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8338071d10 2021-03-09T09:54:40.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010490 con 0x7f8324005e00 2021-03-09T09:54:40.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:54:40.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:40.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:54:40.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:40.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:40.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:40.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:40.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:40.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:40.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e6c0 con 0x7f8338071d10 2021-03-09T09:54:40.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801d6e0 secure :-1 s=READY pgs=5501 cs=0 l=1 rev1=1 rx=0x7f833c05ac20 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:40.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c062350 con 0x7f8338071d10 2021-03-09T09:54:40.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.504+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c010940 con 0x7f8338071d10 2021-03-09T09:54:40.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:40.508+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0572b0 con 0x7f8338071d10 2021-03-09T09:54:40.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:54:40 smithi146 bash[13136]: debug 2021-03-09T09:54:40.500+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15693 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:54:45.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:54:45.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:45.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801d6e0 secure :-1 s=READY pgs=5501 cs=0 l=1 rev1=1 rx=0x7f833c05ac20 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:45.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801d6e0 secure :-1 s=READY pgs=5501 cs=0 l=1 rev1=1 rx=0x7f833c05ac20 tx=0x7f833c04de90).stop 2021-03-09T09:54:45.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801d6e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:45.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801d6e0 unknown :-1 s=CLOSED pgs=5501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:45.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:45.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:45.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:45.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fd80 con 0x7f8324003400 2021-03-09T09:54:45.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f83380091b0 2021-03-09T09:54:45.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008360 con 0x7f8324005e00 2021-03-09T09:54:45.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:54:45.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:45.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:54:45.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:45.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:45.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:45.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:45.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:45.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:45.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801d2b0 con 0x7f83380091b0 2021-03-09T09:54:45.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5510 cs=0 l=1 rev1=1 rx=0x7f833c029890 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:45.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.508+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c045000 con 0x7f83380091b0 2021-03-09T09:54:45.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.512+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01a270 con 0x7f83380091b0 2021-03-09T09:54:45.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:45.516+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01abd0 con 0x7f83380091b0 2021-03-09T09:54:46.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:54:45 smithi146 bash[13136]: debug 2021-03-09T09:54:45.504+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15710 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:54:50.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:54:50.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:50.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5510 cs=0 l=1 rev1=1 rx=0x7f833c029890 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:50.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 secure :-1 s=READY pgs=5510 cs=0 l=1 rev1=1 rx=0x7f833c029890 tx=0x7f833c04de90).stop 2021-03-09T09:54:50.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:50.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:50.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:50.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:50.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:50.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002480 con 0x7f8324003400 2021-03-09T09:54:50.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8338071d10 2021-03-09T09:54:50.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064e10 con 0x7f8324005e00 2021-03-09T09:54:50.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:54:50.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:50.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:54:50.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:50.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.508+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:50.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.512+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:50.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.512+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:50.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.512+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:50.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.512+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:50.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.512+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380084a0 con 0x7f8338071d10 2021-03-09T09:54:50.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.512+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801d6e0 secure :-1 s=READY pgs=5520 cs=0 l=1 rev1=1 rx=0x7f833c049020 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:50.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.512+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c057620 con 0x7f8338071d10 2021-03-09T09:54:50.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.512+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c064000 con 0x7f8338071d10 2021-03-09T09:54:50.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:50.516+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f8338071d10 2021-03-09T09:54:51.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:54:50 smithi146 bash[13136]: debug 2021-03-09T09:54:50.508+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15745 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:54:55.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:54:55.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:55.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801d6e0 secure :-1 s=READY pgs=5520 cs=0 l=1 rev1=1 rx=0x7f833c049020 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:55.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801d6e0 secure :-1 s=READY pgs=5520 cs=0 l=1 rev1=1 rx=0x7f833c049020 tx=0x7f833c04de90).stop 2021-03-09T09:54:55.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801d6e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:55.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801d6e0 unknown :-1 s=CLOSED pgs=5520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:55.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:55.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:55.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:55.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023c00 con 0x7f8324003400 2021-03-09T09:54:55.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f83380091b0 2021-03-09T09:54:55.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d330 con 0x7f8324005e00 2021-03-09T09:54:55.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:54:55.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:55.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:54:55.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:55.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:55.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:55.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:55.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:55.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:55.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380117b0 con 0x7f83380091b0 2021-03-09T09:54:55.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833802b2d0 secure :-1 s=READY pgs=5526 cs=0 l=1 rev1=1 rx=0x7f833c02f930 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:55.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c02f3b0 con 0x7f83380091b0 2021-03-09T09:54:55.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.512+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c057620 con 0x7f83380091b0 2021-03-09T09:54:55.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:54:55.516+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f83380091b0 2021-03-09T09:54:56.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:54:55 smithi146 bash[13136]: debug 2021-03-09T09:54:55.516+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15760 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:55:00.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:55:00.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:00.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833802b2d0 secure :-1 s=READY pgs=5526 cs=0 l=1 rev1=1 rx=0x7f833c02f930 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:00.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833802b2d0 secure :-1 s=READY pgs=5526 cs=0 l=1 rev1=1 rx=0x7f833c02f930 tx=0x7f833c04de90).stop 2021-03-09T09:55:00.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:00.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=5526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:00.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:00.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:00.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:00.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006350 con 0x7f8324003400 2021-03-09T09:55:00.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022030 con 0x7f8338071d10 2021-03-09T09:55:00.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a8d0 con 0x7f8324005e00 2021-03-09T09:55:00.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:55:00.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:00.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:00.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:55:00.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:00.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:00.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:00.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:00.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:00.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017330 con 0x7f8338071d10 2021-03-09T09:55:00.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5534 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:00.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0079d0 con 0x7f8338071d10 2021-03-09T09:55:00.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0077f0 con 0x7f8338071d10 2021-03-09T09:55:00.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:00.519+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c03e9c0 con 0x7f8338071d10 2021-03-09T09:55:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:55:00 smithi146 bash[13136]: debug 2021-03-09T09:55:00.516+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15797 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:55:05.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:55:05.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:05.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5534 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:05.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5534 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c04de90).stop 2021-03-09T09:55:05.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:05.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:05.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:05.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833801e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:05.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:05.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:55:05.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019d70 con 0x7f83380091b0 2021-03-09T09:55:05.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380149c0 con 0x7f8324005e00 2021-03-09T09:55:05.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:55:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:55:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833801e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:05.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:05.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:05.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:05.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:05.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.519+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278f0 con 0x7f83380091b0 2021-03-09T09:55:05.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.523+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833801e0e0 secure :-1 s=READY pgs=5543 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:05.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f83380091b0 2021-03-09T09:55:05.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f83380091b0 2021-03-09T09:55:05.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:05.527+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ab20 con 0x7f83380091b0 2021-03-09T09:55:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:55:05 smithi146 bash[13136]: debug 2021-03-09T09:55:05.520+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15819 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:55:10.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833801e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:55:10.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833801e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:10.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833801e0e0 secure :-1 s=READY pgs=5543 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:10.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833801e0e0 secure :-1 s=READY pgs=5543 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834404c330).stop 2021-03-09T09:55:10.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833801e0e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:10.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833801e0e0 unknown :-1 s=CLOSED pgs=5543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:10.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:10.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:10.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:10.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d4d0 con 0x7f8324003400 2021-03-09T09:55:10.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019320 con 0x7f8338071d10 2021-03-09T09:55:10.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T09:55:10.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:55:10.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:10.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:55:10.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:10.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:10.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:10.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:10.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:10.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:10.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.523+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800fd30 con 0x7f8338071d10 2021-03-09T09:55:10.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.527+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5552 cs=0 l=1 rev1=1 rx=0x7f833c02f520 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:10.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.527+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c057620 con 0x7f8338071d10 2021-03-09T09:55:10.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.527+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01acc0 con 0x7f8338071d10 2021-03-09T09:55:10.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:10.527+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0572b0 con 0x7f8338071d10 2021-03-09T09:55:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:55:10 smithi146 bash[13136]: debug 2021-03-09T09:55:10.523+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15845 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:55:15.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.523+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:55:15.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.523+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:15.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.523+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5552 cs=0 l=1 rev1=1 rx=0x7f833c02f520 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:15.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.523+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5552 cs=0 l=1 rev1=1 rx=0x7f833c02f520 tx=0x7f833c04de90).stop 2021-03-09T09:55:15.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:15.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.523+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:15.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.523+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:15.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.523+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:15.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.523+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833806cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:15.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022030 con 0x7f8324003400 2021-03-09T09:55:15.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010ec0 con 0x7f83380091b0 2021-03-09T09:55:15.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:55:15.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.527+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:55:15.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.527+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:15.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.527+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:55:15.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.527+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:15.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.527+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833806cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:15.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.527+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:15.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.527+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:15.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.527+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:15.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.527+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:15.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.527+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800fd30 con 0x7f83380091b0 2021-03-09T09:55:15.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.527+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833806cd70 secure :-1 s=READY pgs=5560 cs=0 l=1 rev1=1 rx=0x7f833c036ad0 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:15.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c036700 con 0x7f83380091b0 2021-03-09T09:55:15.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f83380091b0 2021-03-09T09:55:15.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:15.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054d30 con 0x7f83380091b0 2021-03-09T09:55:16.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:55:15 smithi146 bash[13136]: debug 2021-03-09T09:55:15.527+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15873 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:55:20.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833806cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:55:20.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833806cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:20.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833806cd70 secure :-1 s=READY pgs=5560 cs=0 l=1 rev1=1 rx=0x7f833c036ad0 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:20.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833806cd70 secure :-1 s=READY pgs=5560 cs=0 l=1 rev1=1 rx=0x7f833c036ad0 tx=0x7f833c04de90).stop 2021-03-09T09:55:20.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833806cd70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:20.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833806cd70 unknown :-1 s=CLOSED pgs=5560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:20.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:20.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:20.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:20.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324003400 2021-03-09T09:55:20.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8338071d10 2021-03-09T09:55:20.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:55:20.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023c00 con 0x7f8324005e00 2021-03-09T09:55:20.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:20.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:55:20.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:20.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.527+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:20.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.531+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:20.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.531+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:20.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.531+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:20.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.531+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:20.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.531+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801d330 con 0x7f8338071d10 2021-03-09T09:55:20.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.531+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5569 cs=0 l=1 rev1=1 rx=0x7f833c02f000 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:20.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c057490 con 0x7f8338071d10 2021-03-09T09:55:20.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0572b0 con 0x7f8338071d10 2021-03-09T09:55:20.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:20.535+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c060260 con 0x7f8338071d10 2021-03-09T09:55:21.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:55:20 smithi146 bash[13136]: debug 2021-03-09T09:55:20.531+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15899 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:55:25.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.531+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:55:25.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.531+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:25.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.531+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5569 cs=0 l=1 rev1=1 rx=0x7f833c02f000 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:25.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.531+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5569 cs=0 l=1 rev1=1 rx=0x7f833c02f000 tx=0x7f833c04de90).stop 2021-03-09T09:55:25.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:25.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.531+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:25.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.531+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:25.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.531+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:25.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.531+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:25.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d8b0 con 0x7f8324003400 2021-03-09T09:55:25.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f83380091b0 2021-03-09T09:55:25.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fb90 con 0x7f8324005e00 2021-03-09T09:55:25.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.531+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:55:25.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.531+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:55:25.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.531+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:25.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.531+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:25.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.535+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:25.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.535+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:25.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.535+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:25.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.535+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:25.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.535+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:25.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.535+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380155c0 con 0x7f83380091b0 2021-03-09T09:55:25.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.535+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5580 cs=0 l=1 rev1=1 rx=0x7f833c0457b0 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:25.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.539+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c02e350 con 0x7f83380091b0 2021-03-09T09:55:25.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.539+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0572b0 con 0x7f83380091b0 2021-03-09T09:55:25.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:25.543+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c077a90 con 0x7f83380091b0 2021-03-09T09:55:26.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:55:25 smithi146 bash[13136]: debug 2021-03-09T09:55:25.531+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15924 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:55:30.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:55:30.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:30.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5580 cs=0 l=1 rev1=1 rx=0x7f833c0457b0 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:30.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5580 cs=0 l=1 rev1=1 rx=0x7f833c0457b0 tx=0x7f833c01af80).stop 2021-03-09T09:55:30.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:30.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=CLOSED pgs=5580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:30.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:30.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:30.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:30.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064e10 con 0x7f8324003400 2021-03-09T09:55:30.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022da0 con 0x7f8338071d10 2021-03-09T09:55:30.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8324005e00 2021-03-09T09:55:30.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:55:30.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:30.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:55:30.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:30.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:30.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:30.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.535+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:30.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.539+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:30.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.539+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:30.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.539+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e6c0 con 0x7f8338071d10 2021-03-09T09:55:30.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.539+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5589 cs=0 l=1 rev1=1 rx=0x7f834400f370 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:30.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.539+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344032d00 con 0x7f8338071d10 2021-03-09T09:55:30.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.543+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T09:55:30.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:30.547+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404c8f0 con 0x7f8338071d10 2021-03-09T09:55:31.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:55:30 smithi146 bash[13136]: debug 2021-03-09T09:55:30.535+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15944 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:55:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.535+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:55:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.535+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.535+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5589 cs=0 l=1 rev1=1 rx=0x7f834400f370 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5589 cs=0 l=1 rev1=1 rx=0x7f834400f370 tx=0x7f834404c330).stop 2021-03-09T09:55:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019320 con 0x7f8324003400 2021-03-09T09:55:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010ec0 con 0x7f83380091b0 2021-03-09T09:55:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:55:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:55:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:55:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.539+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f83380091b0 2021-03-09T09:55:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.543+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5597 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.543+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0294a0 con 0x7f83380091b0 2021-03-09T09:55:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.543+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f83380091b0 2021-03-09T09:55:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:35.547+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c077a90 con 0x7f83380091b0 2021-03-09T09:55:36.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:55:35 smithi146 bash[13136]: debug 2021-03-09T09:55:35.539+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 15975 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:55:40.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.539+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:55:40.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.539+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:40.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.539+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5597 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:40.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.539+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5597 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c01af80).stop 2021-03-09T09:55:40.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.539+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:40.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.539+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=CLOSED pgs=5597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:40.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.539+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:40.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.539+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:40.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.539+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:40.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.539+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066ac0 con 0x7f8324003400 2021-03-09T09:55:40.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.539+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fb90 con 0x7f8338071d10 2021-03-09T09:55:40.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.539+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022da0 con 0x7f8324005e00 2021-03-09T09:55:40.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.543+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:40.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.543+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:55:40.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.543+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:40.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.543+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:55:40.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.543+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:40.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.551+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:40.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.551+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:40.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.551+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:40.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.551+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:40.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.551+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801a880 con 0x7f8338071d10 2021-03-09T09:55:40.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.551+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5607 cs=0 l=1 rev1=1 rx=0x7f833801df70 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:40.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.551+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401f120 con 0x7f8338071d10 2021-03-09T09:55:40.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.551+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T09:55:40.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:40.551+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402b220 con 0x7f8338071d10 2021-03-09T09:55:41.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:55:40 smithi146 bash[13136]: debug 2021-03-09T09:55:40.543+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16001 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:55:45.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:55:45.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:45.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5607 cs=0 l=1 rev1=1 rx=0x7f833801df70 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:45.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5607 cs=0 l=1 rev1=1 rx=0x7f833801df70 tx=0x7f834404c330).stop 2021-03-09T09:55:45.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:45.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:45.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:45.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:45.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:45.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024730 con 0x7f8324003400 2021-03-09T09:55:45.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f83380091b0 2021-03-09T09:55:45.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:55:45.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:55:45.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:45.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:55:45.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:45.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.543+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:45.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.547+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:45.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.547+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:45.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.547+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:45.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.547+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:45.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.547+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c290 con 0x7f83380091b0 2021-03-09T09:55:45.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.547+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5612 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:45.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.547+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c062350 con 0x7f83380091b0 2021-03-09T09:55:45.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.547+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c045000 con 0x7f83380091b0 2021-03-09T09:55:45.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:45.551+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c077a90 con 0x7f83380091b0 2021-03-09T09:55:46.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:55:45 smithi146 bash[13136]: debug 2021-03-09T09:55:45.543+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16026 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:55:50.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:55:50.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:50.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5612 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:50.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5612 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01af80).stop 2021-03-09T09:55:50.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:50.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=CLOSED pgs=5612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:50.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:50.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:50.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:50.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:55:50.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8338071d10 2021-03-09T09:55:50.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064e10 con 0x7f8324005e00 2021-03-09T09:55:50.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:55:50.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:50.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:55:50.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:50.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:50.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:50.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:50.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:50.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:50.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.547+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010e70 con 0x7f8338071d10 2021-03-09T09:55:50.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.551+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5621 cs=0 l=1 rev1=1 rx=0x7f833801edf0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:50.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.551+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344010df0 con 0x7f8338071d10 2021-03-09T09:55:50.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.551+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T09:55:50.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:50.555+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834401f670 con 0x7f8338071d10 2021-03-09T09:55:51.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:55:50 smithi146 bash[13136]: debug 2021-03-09T09:55:50.547+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16048 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:55:55.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.546+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:55:55.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.546+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:55.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.546+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5621 cs=0 l=1 rev1=1 rx=0x7f833801edf0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.546+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5621 cs=0 l=1 rev1=1 rx=0x7f833801edf0 tx=0x7f834404c330).stop 2021-03-09T09:55:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.546+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.546+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.546+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.546+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.546+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.546+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8324003400 2021-03-09T09:55:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.546+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f83380091b0 2021-03-09T09:55:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.546+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:55:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.550+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:55:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.550+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.550+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:55:55.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.550+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:55.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.550+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:55.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.550+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:55.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.550+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:55.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.550+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:55.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.550+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:55.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.550+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028cf0 con 0x7f83380091b0 2021-03-09T09:55:55.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.550+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5630 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:55.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.554+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401f330 con 0x7f83380091b0 2021-03-09T09:55:55.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.554+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f83380091b0 2021-03-09T09:55:55.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:55:55.558+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344078190 con 0x7f83380091b0 2021-03-09T09:55:56.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:55:55 smithi146 bash[13136]: debug 2021-03-09T09:55:55.551+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16077 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:56:00.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.550+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:56:00.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.550+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:00.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.550+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5630 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:00.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.550+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5630 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834404c330).stop 2021-03-09T09:56:00.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.550+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:00.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.550+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=CLOSED pgs=5630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:00.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.550+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:00.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.550+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:00.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.550+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:00.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.550+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020710 con 0x7f8324003400 2021-03-09T09:56:00.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.550+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014200 con 0x7f8338071d10 2021-03-09T09:56:00.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.550+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020560 con 0x7f8324005e00 2021-03-09T09:56:00.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.550+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:56:00.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.550+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:00.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.550+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:56:00.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.550+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:00.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.554+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:00.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.554+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:00.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.554+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:00.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.554+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:00.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.554+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:00.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.554+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f8338071d10 2021-03-09T09:56:00.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.558+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5640 cs=0 l=1 rev1=1 rx=0x7f8338002a30 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:00.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.558+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c077a90 con 0x7f8338071d10 2021-03-09T09:56:00.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f8338071d10 2021-03-09T09:56:00.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:00.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338071d10 2021-03-09T09:56:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:56:00 smithi146 bash[13136]: debug 2021-03-09T09:56:00.555+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16102 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:56:05.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:56:05.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:05.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5640 cs=0 l=1 rev1=1 rx=0x7f8338002a30 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:05.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5640 cs=0 l=1 rev1=1 rx=0x7f8338002a30 tx=0x7f833c01af80).stop 2021-03-09T09:56:05.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:05.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:05.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:05.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:05.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:05.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:56:05.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062380 con 0x7f83380091b0 2021-03-09T09:56:05.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8324005e00 2021-03-09T09:56:05.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:56:05.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:05.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:56:05.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:05.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.554+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:05.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.558+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:05.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.558+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:05.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.558+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:05.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.558+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:05.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.558+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338006300 con 0x7f83380091b0 2021-03-09T09:56:05.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.558+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5647 cs=0 l=1 rev1=1 rx=0x7f833c091670 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:05.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05fe60 con 0x7f83380091b0 2021-03-09T09:56:05.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05fe60 con 0x7f83380091b0 2021-03-09T09:56:05.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:05.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c091ba0 con 0x7f83380091b0 2021-03-09T09:56:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:56:05 smithi146 bash[13136]: debug 2021-03-09T09:56:05.555+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16128 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:56:10.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:56:10.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:10.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5647 cs=0 l=1 rev1=1 rx=0x7f833c091670 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:10.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5647 cs=0 l=1 rev1=1 rx=0x7f833c091670 tx=0x7f833c01af80).stop 2021-03-09T09:56:10.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:10.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=CLOSED pgs=5647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:10.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:10.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:10.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:10.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d310 con 0x7f8324003400 2021-03-09T09:56:10.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014200 con 0x7f8338071d10 2021-03-09T09:56:10.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017cc0 con 0x7f8324005e00 2021-03-09T09:56:10.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:56:10.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:10.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:10.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:56:10.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.558+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:10.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.562+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:10.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.562+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:10.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.562+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:10.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.562+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:10.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.562+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801d8d0 con 0x7f8338071d10 2021-03-09T09:56:10.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.562+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5658 cs=0 l=1 rev1=1 rx=0x7f833c02fc00 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:10.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0647e0 con 0x7f8338071d10 2021-03-09T09:56:10.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c042380 con 0x7f8338071d10 2021-03-09T09:56:10.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:10.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0118e0 con 0x7f8338071d10 2021-03-09T09:56:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:56:10 smithi146 bash[13136]: debug 2021-03-09T09:56:10.559+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16147 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:56:15.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:56:15.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:15.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5658 cs=0 l=1 rev1=1 rx=0x7f833c02fc00 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:15.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5658 cs=0 l=1 rev1=1 rx=0x7f833c02fc00 tx=0x7f833c01af80).stop 2021-03-09T09:56:15.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:15.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:15.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:15.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:15.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:15.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:56:15.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007290 con 0x7f83380091b0 2021-03-09T09:56:15.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8324005e00 2021-03-09T09:56:15.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:56:15.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:15.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:56:15.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:15.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:15.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:15.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:15.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:15.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:15.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c290 con 0x7f83380091b0 2021-03-09T09:56:15.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.562+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5664 cs=0 l=1 rev1=1 rx=0x7f834402b700 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:15.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402baa0 con 0x7f83380091b0 2021-03-09T09:56:15.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f83380091b0 2021-03-09T09:56:15.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:15.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402b730 con 0x7f83380091b0 2021-03-09T09:56:16.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:56:15 smithi146 bash[13136]: debug 2021-03-09T09:56:15.562+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16179 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:56:20.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.562+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:56:20.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.562+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:20.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.562+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5664 cs=0 l=1 rev1=1 rx=0x7f834402b700 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:20.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.562+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5664 cs=0 l=1 rev1=1 rx=0x7f834402b700 tx=0x7f834404c330).stop 2021-03-09T09:56:20.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:20.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.562+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=CLOSED pgs=5664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:20.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.562+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:20.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.562+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:20.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.562+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:20.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.562+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:20.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063420 con 0x7f8324003400 2021-03-09T09:56:20.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d9c0 con 0x7f8338071d10 2021-03-09T09:56:20.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d310 con 0x7f8324005e00 2021-03-09T09:56:20.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.562+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:56:20.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:20.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.566+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:56:20.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.566+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:20.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:20.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:20.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:20.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:20.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338025a20 con 0x7f8338071d10 2021-03-09T09:56:20.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5673 cs=0 l=1 rev1=1 rx=0x7f833c072700 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:20.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c072560 con 0x7f8338071d10 2021-03-09T09:56:20.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c4d0 con 0x7f8338071d10 2021-03-09T09:56:20.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:20.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338071d10 2021-03-09T09:56:21.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:56:20 smithi146 bash[13136]: debug 2021-03-09T09:56:20.566+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16204 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:56:25.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:56:25.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:25.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5673 cs=0 l=1 rev1=1 rx=0x7f833c072700 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:25.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5673 cs=0 l=1 rev1=1 rx=0x7f833c072700 tx=0x7f833c01af80).stop 2021-03-09T09:56:25.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:25.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:25.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:25.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:25.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:25.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017cc0 con 0x7f8324003400 2021-03-09T09:56:25.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f83380091b0 2021-03-09T09:56:25.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063420 con 0x7f8324005e00 2021-03-09T09:56:25.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:56:25.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:25.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:56:25.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:25.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:25.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:25.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:25.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:25.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:25.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278f0 con 0x7f83380091b0 2021-03-09T09:56:25.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5685 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:25.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c02fe40 con 0x7f83380091b0 2021-03-09T09:56:25.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c4d0 con 0x7f83380091b0 2021-03-09T09:56:25.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:25.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c064000 con 0x7f83380091b0 2021-03-09T09:56:26.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:56:25 smithi146 bash[13136]: debug 2021-03-09T09:56:25.566+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16230 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:56:30.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:56:30.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:30.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5685 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:30.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5685 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c01af80).stop 2021-03-09T09:56:30.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:30.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=CLOSED pgs=5685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:30.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:30.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:30.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:30.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016b10 con 0x7f8324003400 2021-03-09T09:56:30.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d9c0 con 0x7f8338071d10 2021-03-09T09:56:30.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d310 con 0x7f8324005e00 2021-03-09T09:56:30.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:56:30.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:56:30.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:30.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:30.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:30.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:30.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:30.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:30.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:30.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c290 con 0x7f8338071d10 2021-03-09T09:56:30.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5692 cs=0 l=1 rev1=1 rx=0x7f833c0573c0 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:30.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00ced0 con 0x7f8338071d10 2021-03-09T09:56:30.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00ced0 con 0x7f8338071d10 2021-03-09T09:56:30.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:30.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01d3a0 con 0x7f8338071d10 2021-03-09T09:56:31.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:56:30 smithi146 bash[13136]: debug 2021-03-09T09:56:30.570+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16255 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:56:35.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:56:35.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:35.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5692 cs=0 l=1 rev1=1 rx=0x7f833c0573c0 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:35.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5692 cs=0 l=1 rev1=1 rx=0x7f833c0573c0 tx=0x7f833c01af80).stop 2021-03-09T09:56:35.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:35.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:35.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:35.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:35.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:35.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:56:35.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f83380091b0 2021-03-09T09:56:35.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066e70 con 0x7f8324005e00 2021-03-09T09:56:35.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:56:35.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:35.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:56:35.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:35.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:35.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:35.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:35.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:35.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:35.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062330 con 0x7f83380091b0 2021-03-09T09:56:35.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.574+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5699 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:35.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f83380091b0 2021-03-09T09:56:35.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033030 con 0x7f83380091b0 2021-03-09T09:56:35.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:35.582+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402bc70 con 0x7f83380091b0 2021-03-09T09:56:36.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:56:35 smithi146 bash[13136]: debug 2021-03-09T09:56:35.574+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16281 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:56:40.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.574+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:56:40.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.574+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:40.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.574+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5699 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:40.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.574+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5699 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834404c330).stop 2021-03-09T09:56:40.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:40.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=CLOSED pgs=5699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:40.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:40.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.578+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:40.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.578+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:40.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016b10 con 0x7f8324003400 2021-03-09T09:56:40.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d9c0 con 0x7f8338071d10 2021-03-09T09:56:40.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.578+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:56:40.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:56:40.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.578+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:40.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.578+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:56:40.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.578+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:40.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.578+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:40.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.578+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:40.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.578+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:40.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.578+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:40.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.578+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:40.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.578+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f8338071d10 2021-03-09T09:56:40.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.578+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5709 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:40.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.582+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401ee20 con 0x7f8338071d10 2021-03-09T09:56:40.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.582+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033040 con 0x7f8338071d10 2021-03-09T09:56:40.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:40.586+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400cac0 con 0x7f8338071d10 2021-03-09T09:56:41.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:56:40 smithi146 bash[13136]: debug 2021-03-09T09:56:40.578+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16304 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:56:45.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:56:45.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:45.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5709 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:45.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5709 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834404c330).stop 2021-03-09T09:56:45.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:45.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:45.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:45.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:45.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:45.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018b10 con 0x7f8324003400 2021-03-09T09:56:45.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064e10 con 0x7f83380091b0 2021-03-09T09:56:45.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009dd0 con 0x7f8324005e00 2021-03-09T09:56:45.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:56:45.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:45.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:56:45.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:45.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:45.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:45.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:45.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:45.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:45.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e680 con 0x7f83380091b0 2021-03-09T09:56:45.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5716 cs=0 l=1 rev1=1 rx=0x7f833800d8a0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:45.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f83380091b0 2021-03-09T09:56:45.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033040 con 0x7f83380091b0 2021-03-09T09:56:45.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:45.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344072190 con 0x7f83380091b0 2021-03-09T09:56:46.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:56:45 smithi146 bash[13136]: debug 2021-03-09T09:56:45.582+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16332 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:56:50.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:56:50.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:50.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5716 cs=0 l=1 rev1=1 rx=0x7f833800d8a0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:50.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5716 cs=0 l=1 rev1=1 rx=0x7f833800d8a0 tx=0x7f834404c330).stop 2021-03-09T09:56:50.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:50.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=CLOSED pgs=5716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:50.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:50.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:50.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:50.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:56:50.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007250 con 0x7f8338071d10 2021-03-09T09:56:50.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b40 con 0x7f8324005e00 2021-03-09T09:56:50.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:56:50.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:50.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:56:50.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:50.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:50.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:50.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:50.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:50.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:50.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380690f0 con 0x7f8338071d10 2021-03-09T09:56:50.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5725 cs=0 l=1 rev1=1 rx=0x7f8344010f90 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:50.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T09:56:50.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T09:56:50.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:50.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834401eac0 con 0x7f8338071d10 2021-03-09T09:56:51.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:56:50 smithi146 bash[13136]: debug 2021-03-09T09:56:50.582+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16357 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:56:55.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:56:55.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:55.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5725 cs=0 l=1 rev1=1 rx=0x7f8344010f90 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:55.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5725 cs=0 l=1 rev1=1 rx=0x7f8344010f90 tx=0x7f834404c330).stop 2021-03-09T09:56:55.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:55.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:55.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:55.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:55.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:55.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:56:55.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016b10 con 0x7f8324003400 2021-03-09T09:56:55.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:55.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:56:55.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:55.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018b10 con 0x7f83380091b0 2021-03-09T09:56:55.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010840 con 0x7f8324005e00 2021-03-09T09:56:55.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:55.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.589+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:55.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.589+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:55.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.589+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:55.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.589+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:55.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.589+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f83380091b0 2021-03-09T09:56:55.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.589+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5736 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:55.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401f120 con 0x7f83380091b0 2021-03-09T09:56:55.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f83380091b0 2021-03-09T09:56:55.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:56:55.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405c120 con 0x7f83380091b0 2021-03-09T09:56:56.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:56:55 smithi146 bash[13136]: debug 2021-03-09T09:56:55.586+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16380 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:57:00.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:57:00.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:00.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5736 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:00.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5736 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834404c330).stop 2021-03-09T09:57:00.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:00.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=CLOSED pgs=5736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:00.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:00.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:00.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:00.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d9c0 con 0x7f8324003400 2021-03-09T09:57:00.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8338071d10 2021-03-09T09:57:00.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012880 con 0x7f8324005e00 2021-03-09T09:57:00.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:57:00.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:00.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:57:00.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:00.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:00.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:00.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:00.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:00.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:00.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.589+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e4e0 con 0x7f8338071d10 2021-03-09T09:57:00.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5743 cs=0 l=1 rev1=1 rx=0x7f833c042b30 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:00.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f8338071d10 2021-03-09T09:57:00.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c4d0 con 0x7f8338071d10 2021-03-09T09:57:00.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:00.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0912d0 con 0x7f8338071d10 2021-03-09T09:57:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:57:00 smithi146 bash[13136]: debug 2021-03-09T09:57:00.590+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16404 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:57:05.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.589+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:57:05.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.589+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:05.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5743 cs=0 l=1 rev1=1 rx=0x7f833c042b30 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:05.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5743 cs=0 l=1 rev1=1 rx=0x7f833c042b30 tx=0x7f833c01af80).stop 2021-03-09T09:57:05.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:05.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:05.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:05.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:05.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:05.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014e90 con 0x7f8324003400 2021-03-09T09:57:05.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bb80 con 0x7f83380091b0 2021-03-09T09:57:05.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b40 con 0x7f8324005e00 2021-03-09T09:57:05.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:57:05.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:57:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.593+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:05.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:05.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.593+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:05.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:05.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.593+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010840 con 0x7f83380091b0 2021-03-09T09:57:05.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5751 cs=0 l=1 rev1=1 rx=0x7f8344017790 tx=0x7f8344072ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:05.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f83380091b0 2021-03-09T09:57:05.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033040 con 0x7f83380091b0 2021-03-09T09:57:05.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:05.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344027d40 con 0x7f83380091b0 2021-03-09T09:57:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:57:05 smithi146 bash[13136]: debug 2021-03-09T09:57:05.594+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16429 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:57:10.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:57:10.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5751 cs=0 l=1 rev1=1 rx=0x7f8344017790 tx=0x7f8344072ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 secure :-1 s=READY pgs=5751 cs=0 l=1 rev1=1 rx=0x7f8344017790 tx=0x7f8344072ea0).stop 2021-03-09T09:57:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f833800f2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f833800f2b0 unknown :-1 s=CLOSED pgs=5751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833802e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016b10 con 0x7f8324003400 2021-03-09T09:57:10.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d9c0 con 0x7f8338071d10 2021-03-09T09:57:10.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062580 con 0x7f8324005e00 2021-03-09T09:57:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:57:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:57:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833802e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.597+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.597+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.597+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f8338071d10 2021-03-09T09:57:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833802e010 secure :-1 s=READY pgs=5760 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344072ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440171c0 con 0x7f8338071d10 2021-03-09T09:57:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T09:57:10.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:10.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f8338071d10 2021-03-09T09:57:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:57:10 smithi146 bash[13136]: debug 2021-03-09T09:57:10.594+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16453 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:57:15.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833802e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:57:15.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833802e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:15.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833802e010 secure :-1 s=READY pgs=5760 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344072ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:15.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833802e010 secure :-1 s=READY pgs=5760 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344072ea0).stop 2021-03-09T09:57:15.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833802e010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:15.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833802e010 unknown :-1 s=CLOSED pgs=5760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:15.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:15.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:15.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:15.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061130 con 0x7f8324003400 2021-03-09T09:57:15.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e870 con 0x7f83380091b0 2021-03-09T09:57:15.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:57:15.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:57:15.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:15.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:57:15.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:15.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.597+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:15.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.601+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:15.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.601+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:15.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.601+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:15.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.601+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:15.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.601+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801f930 con 0x7f83380091b0 2021-03-09T09:57:15.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.601+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5768 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:15.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01c4d0 con 0x7f83380091b0 2021-03-09T09:57:15.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f83380091b0 2021-03-09T09:57:15.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:15.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f83380091b0 2021-03-09T09:57:16.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:57:15 smithi146 bash[13136]: debug 2021-03-09T09:57:15.598+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16477 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:57:20.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:57:20.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:20.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5768 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:20.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5768 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c01af80).stop 2021-03-09T09:57:20.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:20.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=CLOSED pgs=5768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:20.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:20.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:20.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:57:20.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:20.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:20.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d9c0 con 0x7f8324003400 2021-03-09T09:57:20.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8338071d10 2021-03-09T09:57:20.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380122c0 con 0x7f8324005e00 2021-03-09T09:57:20.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:57:20.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:20.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:20.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:20.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:20.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.601+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:20.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.605+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:20.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.605+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017550 con 0x7f8338071d10 2021-03-09T09:57:20.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.605+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5777 cs=0 l=1 rev1=1 rx=0x7f833c0643a0 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:20.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0369c0 con 0x7f8338071d10 2021-03-09T09:57:20.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f8338071d10 2021-03-09T09:57:20.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:20.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01abd0 con 0x7f8338071d10 2021-03-09T09:57:21.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:57:20 smithi146 bash[13136]: debug 2021-03-09T09:57:20.601+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16506 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:57:25.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:57:25.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:25.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5777 cs=0 l=1 rev1=1 rx=0x7f833c0643a0 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:25.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5777 cs=0 l=1 rev1=1 rx=0x7f833c0643a0 tx=0x7f833c01af80).stop 2021-03-09T09:57:25.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:25.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:25.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:25.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:25.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:57:25.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:25.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:25.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e870 con 0x7f8324003400 2021-03-09T09:57:25.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f83380091b0 2021-03-09T09:57:25.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d9c0 con 0x7f8324005e00 2021-03-09T09:57:25.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:57:25.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:25.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:25.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:25.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:25.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:25.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:25.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338016b10 con 0x7f83380091b0 2021-03-09T09:57:25.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.605+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5786 cs=0 l=1 rev1=1 rx=0x7f83440499b0 tx=0x7f8344072ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:25.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401e3c0 con 0x7f83380091b0 2021-03-09T09:57:25.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033040 con 0x7f83380091b0 2021-03-09T09:57:25.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:25.613+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f83380091b0 2021-03-09T09:57:26.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:57:25 smithi146 bash[13136]: debug 2021-03-09T09:57:25.605+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16531 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:57:30.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:57:30.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:30.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5786 cs=0 l=1 rev1=1 rx=0x7f83440499b0 tx=0x7f8344072ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:30.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5786 cs=0 l=1 rev1=1 rx=0x7f83440499b0 tx=0x7f8344072ea0).stop 2021-03-09T09:57:30.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:30.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=CLOSED pgs=5786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:30.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:30.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:30.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:30.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005550 con 0x7f8324003400 2021-03-09T09:57:30.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380122c0 con 0x7f8338071d10 2021-03-09T09:57:30.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068120 con 0x7f8324005e00 2021-03-09T09:57:30.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:57:30.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:57:30.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:30.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:30.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:30.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:30.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:30.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:30.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:30.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017550 con 0x7f8338071d10 2021-03-09T09:57:30.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.609+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5795 cs=0 l=1 rev1=1 rx=0x7f8338020670 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:30.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.613+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c007bc0 con 0x7f8338071d10 2021-03-09T09:57:30.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.613+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c4d0 con 0x7f8338071d10 2021-03-09T09:57:30.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:30.617+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f8338071d10 2021-03-09T09:57:31.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:57:30 smithi146 bash[13136]: debug 2021-03-09T09:57:30.609+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16557 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:57:35.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.608+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:57:35.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.608+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:35.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.608+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5795 cs=0 l=1 rev1=1 rx=0x7f8338020670 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:35.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.608+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5795 cs=0 l=1 rev1=1 rx=0x7f8338020670 tx=0x7f833c01af80).stop 2021-03-09T09:57:35.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:35.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.608+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:35.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.608+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:35.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.608+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:35.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.608+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:35.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d9c0 con 0x7f8324003400 2021-03-09T09:57:35.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012310 con 0x7f83380091b0 2021-03-09T09:57:35.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064e10 con 0x7f8324005e00 2021-03-09T09:57:35.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.608+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:57:35.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.608+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:35.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.608+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:57:35.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.608+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:35.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.612+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:35.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.612+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:35.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.612+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:35.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.612+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:35.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.612+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:35.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.612+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278f0 con 0x7f83380091b0 2021-03-09T09:57:35.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.612+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5803 cs=0 l=1 rev1=1 rx=0x7f833c0187d0 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:35.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.616+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c041680 con 0x7f83380091b0 2021-03-09T09:57:35.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.616+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c064100 con 0x7f83380091b0 2021-03-09T09:57:35.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:35.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c026900 con 0x7f83380091b0 2021-03-09T09:57:36.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:57:35 smithi146 bash[13136]: debug 2021-03-09T09:57:35.613+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16581 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:57:40.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:57:40.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:40.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5803 cs=0 l=1 rev1=1 rx=0x7f833c0187d0 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:40.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5803 cs=0 l=1 rev1=1 rx=0x7f833c0187d0 tx=0x7f833c01af80).stop 2021-03-09T09:57:40.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:40.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=CLOSED pgs=5803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:40.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:40.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:40.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:40.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f9d0 con 0x7f8324003400 2021-03-09T09:57:40.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8338071d10 2021-03-09T09:57:40.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380682d0 con 0x7f8324005e00 2021-03-09T09:57:40.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:57:40.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:57:40.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:40.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:40.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.612+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:40.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.616+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:40.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.616+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:40.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.616+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:40.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.616+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:40.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.616+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c290 con 0x7f8338071d10 2021-03-09T09:57:40.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.616+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5812 cs=0 l=1 rev1=1 rx=0x7f833c057cf0 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:40.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c064060 con 0x7f8338071d10 2021-03-09T09:57:40.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0641c0 con 0x7f8338071d10 2021-03-09T09:57:40.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:40.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c041b50 con 0x7f8338071d10 2021-03-09T09:57:41.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:57:40 smithi146 bash[13136]: debug 2021-03-09T09:57:40.613+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16608 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:57:45.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:57:45.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:45.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5812 cs=0 l=1 rev1=1 rx=0x7f833c057cf0 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:45.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5812 cs=0 l=1 rev1=1 rx=0x7f833c057cf0 tx=0x7f833c01af80).stop 2021-03-09T09:57:45.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:45.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:45.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:45.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:45.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:45.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T09:57:45.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d9c0 con 0x7f83380091b0 2021-03-09T09:57:45.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d4f0 con 0x7f8324005e00 2021-03-09T09:57:45.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:57:45.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:45.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:57:45.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:45.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.616+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:45.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:45.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:45.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:45.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:45.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017550 con 0x7f83380091b0 2021-03-09T09:57:45.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5821 cs=0 l=1 rev1=1 rx=0x7f833c01c640 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:45.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c061670 con 0x7f83380091b0 2021-03-09T09:57:45.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c042d70 con 0x7f83380091b0 2021-03-09T09:57:45.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:45.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02fc90 con 0x7f83380091b0 2021-03-09T09:57:46.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:57:45 smithi146 bash[13136]: debug 2021-03-09T09:57:45.617+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16632 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:57:50.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:57:50.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:50.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5821 cs=0 l=1 rev1=1 rx=0x7f833c01c640 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:50.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5821 cs=0 l=1 rev1=1 rx=0x7f833c01c640 tx=0x7f833c01af80).stop 2021-03-09T09:57:50.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:50.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=CLOSED pgs=5821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:50.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:50.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:50.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:50.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007250 con 0x7f8324003400 2021-03-09T09:57:50.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380680e0 con 0x7f8338071d10 2021-03-09T09:57:50.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380682d0 con 0x7f8324005e00 2021-03-09T09:57:50.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:57:50.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:50.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:57:50.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:50.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:50.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:50.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:50.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:50.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:50.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278f0 con 0x7f8338071d10 2021-03-09T09:57:50.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.624+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5829 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:50.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338071d10 2021-03-09T09:57:50.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d2d0 con 0x7f8338071d10 2021-03-09T09:57:50.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:50.628+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018320 con 0x7f8338071d10 2021-03-09T09:57:51.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:57:50 smithi146 bash[13136]: debug 2021-03-09T09:57:50.621+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16659 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:57:55.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:57:55.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:55.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5829 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:55.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5829 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01af80).stop 2021-03-09T09:57:55.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:55.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:55.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:57:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:55.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f340 con 0x7f8324003400 2021-03-09T09:57:55.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f83380091b0 2021-03-09T09:57:55.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010e30 con 0x7f8324005e00 2021-03-09T09:57:55.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:55.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:57:55.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:55.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f83380091b0 2021-03-09T09:57:55.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.624+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5838 cs=0 l=1 rev1=1 rx=0x7f833c007c20 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:55.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.628+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c02f580 con 0x7f83380091b0 2021-03-09T09:57:55.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.628+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c02f6e0 con 0x7f83380091b0 2021-03-09T09:57:55.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:57:55.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02f580 con 0x7f83380091b0 2021-03-09T09:57:56.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:57:55 smithi146 bash[13136]: debug 2021-03-09T09:57:55.625+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16683 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:58:00.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.624+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T09:58:00.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.624+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:00.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.624+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5838 cs=0 l=1 rev1=1 rx=0x7f833c007c20 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:00.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.624+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5838 cs=0 l=1 rev1=1 rx=0x7f833c007c20 tx=0x7f833c01af80).stop 2021-03-09T09:58:00.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:00.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.624+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=CLOSED pgs=5838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:00.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.624+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:00.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.624+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:00.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.624+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:00.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063ca0 con 0x7f8324003400 2021-03-09T09:58:00.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f8338071d10 2021-03-09T09:58:00.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.628+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:58:00.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.628+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:00.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.628+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:00.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028b90 con 0x7f8324005e00 2021-03-09T09:58:00.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.628+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:58:00.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.628+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:00.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.632+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:00.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.632+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:00.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.632+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:00.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.632+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:00.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.632+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8338071d10 2021-03-09T09:58:00.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.632+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5848 cs=0 l=1 rev1=1 rx=0x7f833c064900 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:00.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c064000 con 0x7f8338071d10 2021-03-09T09:58:00.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c072a50 con 0x7f8338071d10 2021-03-09T09:58:00.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:00.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c091d10 con 0x7f8338071d10 2021-03-09T09:58:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:58:00 smithi146 bash[13136]: debug 2021-03-09T09:58:00.629+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16710 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:58:05.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:58:05.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:05.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5848 cs=0 l=1 rev1=1 rx=0x7f833c064900 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:05.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5848 cs=0 l=1 rev1=1 rx=0x7f833c064900 tx=0x7f833c01af80).stop 2021-03-09T09:58:05.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:05.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:05.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:05.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:05.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:05.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d9c0 con 0x7f8324003400 2021-03-09T09:58:05.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f340 con 0x7f83380091b0 2021-03-09T09:58:05.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018320 con 0x7f8324005e00 2021-03-09T09:58:05.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:58:05.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:05.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:58:05.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:05.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:05.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:05.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:05.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:05.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:05.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017550 con 0x7f83380091b0 2021-03-09T09:58:05.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5857 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344072ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:05.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401fec0 con 0x7f83380091b0 2021-03-09T09:58:05.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f83380091b0 2021-03-09T09:58:05.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:05.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404c8f0 con 0x7f83380091b0 2021-03-09T09:58:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:58:05 smithi146 bash[13136]: debug 2021-03-09T09:58:05.633+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16734 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:58:10.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:58:10.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:10.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5857 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344072ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:10.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 secure :-1 s=READY pgs=5857 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344072ea0).stop 2021-03-09T09:58:10.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:10.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338021080 unknown :-1 s=CLOSED pgs=5857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:10.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:10.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:10.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338021dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:10.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002d00 con 0x7f8324003400 2021-03-09T09:58:10.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d2f0 con 0x7f8338071d10 2021-03-09T09:58:10.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007250 con 0x7f8324005e00 2021-03-09T09:58:10.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:58:10.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:58:10.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:10.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:10.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338021dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:10.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:10.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:10.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:10.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:10.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800e2b0 con 0x7f8338071d10 2021-03-09T09:58:10.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338021dc0 secure :-1 s=READY pgs=5864 cs=0 l=1 rev1=1 rx=0x7f833c040940 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:10.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01aca0 con 0x7f8338071d10 2021-03-09T09:58:10.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01aca0 con 0x7f8338071d10 2021-03-09T09:58:10.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:10.640+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c091d10 con 0x7f8338071d10 2021-03-09T09:58:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:58:10 smithi146 bash[13136]: debug 2021-03-09T09:58:10.633+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16758 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:58:15.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338021dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:58:15.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338021dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:15.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338021dc0 secure :-1 s=READY pgs=5864 cs=0 l=1 rev1=1 rx=0x7f833c040940 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:15.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338021dc0 secure :-1 s=READY pgs=5864 cs=0 l=1 rev1=1 rx=0x7f833c040940 tx=0x7f833c01af80).stop 2021-03-09T09:58:15.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338021dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:15.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338021dc0 unknown :-1 s=CLOSED pgs=5864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:15.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338017720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:15.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:15.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:15.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d8c0 con 0x7f8324003400 2021-03-09T09:58:15.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f83380091b0 2021-03-09T09:58:15.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d9c0 con 0x7f8324005e00 2021-03-09T09:58:15.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:58:15.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:15.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:58:15.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:15.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338017720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:15.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f83380091b0 2021-03-09T09:58:15.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.640+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338017720 secure :-1 s=READY pgs=5873 cs=0 l=1 rev1=1 rx=0x7f833c007c20 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:15.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.640+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c045b40 con 0x7f83380091b0 2021-03-09T09:58:15.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.640+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0416f0 con 0x7f83380091b0 2021-03-09T09:58:15.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:15.644+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c045930 con 0x7f83380091b0 2021-03-09T09:58:16.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:58:15 smithi146 bash[13136]: debug 2021-03-09T09:58:15.637+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16785 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:58:20.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338017720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:58:20.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338017720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:20.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338017720 secure :-1 s=READY pgs=5873 cs=0 l=1 rev1=1 rx=0x7f833c007c20 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:20.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338017720 secure :-1 s=READY pgs=5873 cs=0 l=1 rev1=1 rx=0x7f833c007c20 tx=0x7f833c01af80).stop 2021-03-09T09:58:20.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338017720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:20.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338017720 unknown :-1 s=CLOSED pgs=5873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:20.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:20.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:20.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:20.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002d00 con 0x7f8324003400 2021-03-09T09:58:20.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007250 con 0x7f8338071d10 2021-03-09T09:58:20.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011bd0 con 0x7f8324005e00 2021-03-09T09:58:20.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:58:20.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:20.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:58:20.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:20.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:20.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:20.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:20.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:20.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:20.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017550 con 0x7f8338071d10 2021-03-09T09:58:20.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5882 cs=0 l=1 rev1=1 rx=0x7f83440162e0 tx=0x7f8344072ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:20.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.644+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T09:58:20.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033030 con 0x7f8338071d10 2021-03-09T09:58:20.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:20.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f8338071d10 2021-03-09T09:58:21.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:58:20 smithi146 bash[13136]: debug 2021-03-09T09:58:20.641+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16812 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:58:25.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:58:25.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:25.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5882 cs=0 l=1 rev1=1 rx=0x7f83440162e0 tx=0x7f8344072ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:25.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5882 cs=0 l=1 rev1=1 rx=0x7f83440162e0 tx=0x7f8344072ea0).stop 2021-03-09T09:58:25.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:25.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:25.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338063080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:25.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:25.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:25.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380153c0 con 0x7f8324003400 2021-03-09T09:58:25.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f83380091b0 2021-03-09T09:58:25.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380168d0 con 0x7f8324005e00 2021-03-09T09:58:25.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:58:25.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:58:25.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:25.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338063080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:25.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:25.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:25.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:25.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:25.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:25.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e530 con 0x7f83380091b0 2021-03-09T09:58:25.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.643+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338063080 secure :-1 s=READY pgs=5890 cs=0 l=1 rev1=1 rx=0x7f833c02fb60 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:25.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.647+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c041530 con 0x7f83380091b0 2021-03-09T09:58:25.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.647+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0412e0 con 0x7f83380091b0 2021-03-09T09:58:25.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:25.651+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c077a90 con 0x7f83380091b0 2021-03-09T09:58:26.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:58:25 smithi146 bash[13136]: debug 2021-03-09T09:58:25.645+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16836 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:58:30.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.647+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:58:30.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.647+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:30.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.647+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338063080 secure :-1 s=READY pgs=5890 cs=0 l=1 rev1=1 rx=0x7f833c02fb60 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:30.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.647+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338063080 secure :-1 s=READY pgs=5890 cs=0 l=1 rev1=1 rx=0x7f833c02fb60 tx=0x7f833c01af80).stop 2021-03-09T09:58:30.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.647+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338063080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:30.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.647+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338063080 unknown :-1 s=CLOSED pgs=5890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:30.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.647+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:30.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.647+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:30.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.647+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:30.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.647+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010e30 con 0x7f8324003400 2021-03-09T09:58:30.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.647+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014f70 con 0x7f8338071d10 2021-03-09T09:58:30.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.647+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009dd0 con 0x7f8324005e00 2021-03-09T09:58:30.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.647+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:58:30.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.651+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:58:30.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.651+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:30.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.651+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:30.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.651+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:30.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.651+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:30.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.651+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:30.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.651+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:30.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.651+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:30.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.651+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c290 con 0x7f8338071d10 2021-03-09T09:58:30.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.651+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5899 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:30.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.651+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c02fbd0 con 0x7f8338071d10 2021-03-09T09:58:30.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.651+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c077a90 con 0x7f8338071d10 2021-03-09T09:58:30.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:30.659+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d1e0 con 0x7f8338071d10 2021-03-09T09:58:31.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:58:30 smithi146 bash[13136]: debug 2021-03-09T09:58:30.648+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16863 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:58:35.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.651+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:58:35.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.651+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:35.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.651+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5899 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:35.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.651+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5899 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c01af80).stop 2021-03-09T09:58:35.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.651+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:35.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.651+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:35.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.651+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:35.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.651+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:35.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.651+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:35.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.651+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338029030 con 0x7f8324003400 2021-03-09T09:58:35.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.651+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380153c0 con 0x7f83380091b0 2021-03-09T09:58:35.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.651+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069860 con 0x7f8324005e00 2021-03-09T09:58:35.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.651+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:58:35.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.651+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:35.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.655+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:58:35.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.655+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:35.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.655+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:35.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.655+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:35.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.655+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:35.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.655+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:35.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.655+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:35.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.655+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017550 con 0x7f83380091b0 2021-03-09T09:58:35.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.655+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 secure :-1 s=READY pgs=5908 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:35.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.655+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c077a90 con 0x7f83380091b0 2021-03-09T09:58:35.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.655+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c036980 con 0x7f83380091b0 2021-03-09T09:58:35.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:35.659+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054310 con 0x7f83380091b0 2021-03-09T09:58:36.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:58:35 smithi146 bash[13136]: debug 2021-03-09T09:58:35.652+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16887 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:58:40.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:58:40.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:40.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 secure :-1 s=READY pgs=5908 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:40.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 secure :-1 s=READY pgs=5908 cs=0 l=1 rev1=1 rx=0x7f833c025530 tx=0x7f833c01af80).stop 2021-03-09T09:58:40.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f83380249b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:40.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 unknown :-1 s=CLOSED pgs=5908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:40.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:40.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:40.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:40.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010e30 con 0x7f8324003400 2021-03-09T09:58:40.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d8c0 con 0x7f8338071d10 2021-03-09T09:58:40.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bb80 con 0x7f8324005e00 2021-03-09T09:58:40.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:58:40.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:40.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:58:40.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:40.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.655+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:40.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.659+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:40.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.659+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:40.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.659+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:40.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.659+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:40.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.659+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806d620 con 0x7f8338071d10 2021-03-09T09:58:40.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.659+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5917 cs=0 l=1 rev1=1 rx=0x7f83440499b0 tx=0x7f8344072ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:40.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.659+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T09:58:40.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.659+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033040 con 0x7f8338071d10 2021-03-09T09:58:40.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:40.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404c8f0 con 0x7f8338071d10 2021-03-09T09:58:41.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:58:40 smithi146 bash[13136]: debug 2021-03-09T09:58:40.652+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16914 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:58:45.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:58:45.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:45.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5917 cs=0 l=1 rev1=1 rx=0x7f83440499b0 tx=0x7f8344072ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:45.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5917 cs=0 l=1 rev1=1 rx=0x7f83440499b0 tx=0x7f8344072ea0).stop 2021-03-09T09:58:45.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:45.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:45.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:45.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:45.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:45.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ddb0 con 0x7f8324003400 2021-03-09T09:58:45.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380153c0 con 0x7f83380091b0 2021-03-09T09:58:45.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061ee0 con 0x7f8324005e00 2021-03-09T09:58:45.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:58:45.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:45.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:58:45.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:45.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.659+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:45.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:45.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:45.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:45.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:45.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c290 con 0x7f83380091b0 2021-03-09T09:58:45.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 secure :-1 s=READY pgs=5926 cs=0 l=1 rev1=1 rx=0x7f833c036910 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:45.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00ced0 con 0x7f83380091b0 2021-03-09T09:58:45.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f83380091b0 2021-03-09T09:58:45.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:45.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0412e0 con 0x7f83380091b0 2021-03-09T09:58:46.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:58:45 smithi146 bash[13136]: debug 2021-03-09T09:58:45.656+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16938 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:58:50.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:58:50.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:50.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 secure :-1 s=READY pgs=5926 cs=0 l=1 rev1=1 rx=0x7f833c036910 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:50.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 secure :-1 s=READY pgs=5926 cs=0 l=1 rev1=1 rx=0x7f833c036910 tx=0x7f833c01af80).stop 2021-03-09T09:58:50.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f83380249b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:50.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 unknown :-1 s=CLOSED pgs=5926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:50.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:50.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:50.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:50.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324003400 2021-03-09T09:58:50.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380168d0 con 0x7f8338071d10 2021-03-09T09:58:50.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d8c0 con 0x7f8324005e00 2021-03-09T09:58:50.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:58:50.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:50.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:50.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:58:50.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:50.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:50.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:50.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:50.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:50.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a1c0 con 0x7f8338071d10 2021-03-09T09:58:50.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5933 cs=0 l=1 rev1=1 rx=0x7f833c06b040 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:50.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338071d10 2021-03-09T09:58:50.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d2d0 con 0x7f8338071d10 2021-03-09T09:58:50.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:50.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02fbd0 con 0x7f8338071d10 2021-03-09T09:58:51.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:58:50 smithi146 bash[13136]: debug 2021-03-09T09:58:50.660+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16964 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:58:55.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.667+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:58:55.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.667+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:55.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5933 cs=0 l=1 rev1=1 rx=0x7f833c06b040 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:55.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5933 cs=0 l=1 rev1=1 rx=0x7f833c06b040 tx=0x7f833c01af80).stop 2021-03-09T09:58:55.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:55.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:55.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:55.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:55.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:55.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bb80 con 0x7f8324003400 2021-03-09T09:58:55.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ddb0 con 0x7f83380091b0 2021-03-09T09:58:55.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014690 con 0x7f8324005e00 2021-03-09T09:58:55.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.667+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:58:55.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:55.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.667+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:58:55.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:55.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:55.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:55.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:55.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:55.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:55.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017550 con 0x7f83380091b0 2021-03-09T09:58:55.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 secure :-1 s=READY pgs=5947 cs=0 l=1 rev1=1 rx=0x7f834406a0b0 tx=0x7f8344072ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:55.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404b090 con 0x7f83380091b0 2021-03-09T09:58:55.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f83380091b0 2021-03-09T09:58:55.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:58:55.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402bb80 con 0x7f83380091b0 2021-03-09T09:58:56.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:58:55 smithi146 bash[13136]: debug 2021-03-09T09:58:55.664+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 16989 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:59:00.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.667+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:59:00.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.667+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:00.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.667+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 secure :-1 s=READY pgs=5947 cs=0 l=1 rev1=1 rx=0x7f834406a0b0 tx=0x7f8344072ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:00.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.667+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 secure :-1 s=READY pgs=5947 cs=0 l=1 rev1=1 rx=0x7f834406a0b0 tx=0x7f8344072ea0).stop 2021-03-09T09:59:00.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f83380249b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:00.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 unknown :-1 s=CLOSED pgs=5947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:00.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:00.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:00.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:00.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069860 con 0x7f8324003400 2021-03-09T09:59:00.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f8338071d10 2021-03-09T09:59:00.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380168d0 con 0x7f8324005e00 2021-03-09T09:59:00.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.671+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:59:00.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.671+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:00.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.671+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:59:00.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.671+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:00.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:00.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:00.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:00.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:00.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:00.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e530 con 0x7f8338071d10 2021-03-09T09:59:00.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5951 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:00.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401f120 con 0x7f8338071d10 2021-03-09T09:59:00.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834401f280 con 0x7f8338071d10 2021-03-09T09:59:00.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:00.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344077240 con 0x7f8338071d10 2021-03-09T09:59:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:59:00 smithi146 bash[13136]: debug 2021-03-09T09:59:00.668+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17013 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:59:05.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:59:05.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:05.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5951 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:05.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5951 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f8344003f90).stop 2021-03-09T09:59:05.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:05.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:05.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:05.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:05.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:05.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380153c0 con 0x7f8324003400 2021-03-09T09:59:05.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ddb0 con 0x7f83380091b0 2021-03-09T09:59:05.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T09:59:05.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:59:05.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:05.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.671+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:59:05.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.671+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:05.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.675+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:05.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.675+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:05.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.675+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:05.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.675+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:05.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.675+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:05.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.675+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338021760 con 0x7f83380091b0 2021-03-09T09:59:05.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.675+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 secure :-1 s=READY pgs=5959 cs=0 l=1 rev1=1 rx=0x7f833c093fd0 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:05.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f83380091b0 2021-03-09T09:59:05.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0252e0 con 0x7f83380091b0 2021-03-09T09:59:05.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:05.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018320 con 0x7f83380091b0 2021-03-09T09:59:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:59:05 smithi146 bash[13136]: debug 2021-03-09T09:59:05.672+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17040 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:59:10.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:59:10.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:10.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 secure :-1 s=READY pgs=5959 cs=0 l=1 rev1=1 rx=0x7f833c093fd0 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:10.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 secure :-1 s=READY pgs=5959 cs=0 l=1 rev1=1 rx=0x7f833c093fd0 tx=0x7f833c01af80).stop 2021-03-09T09:59:10.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f83380249b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:10.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f83380249b0 unknown :-1 s=CLOSED pgs=5959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:10.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:10.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:10.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:10.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f8324003400 2021-03-09T09:59:10.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8338071d10 2021-03-09T09:59:10.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380628d0 con 0x7f8324005e00 2021-03-09T09:59:10.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:59:10.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:10.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:59:10.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:10.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:10.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.675+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:10.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.679+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:10.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.679+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:10.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.679+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:10.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.679+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806df10 con 0x7f8338071d10 2021-03-09T09:59:10.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.679+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5968 cs=0 l=1 rev1=1 rx=0x7f833c040e50 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:10.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0451e0 con 0x7f8338071d10 2021-03-09T09:59:10.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c045000 con 0x7f8338071d10 2021-03-09T09:59:10.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:10.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c079440 con 0x7f8338071d10 2021-03-09T09:59:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:59:10 smithi146 bash[13136]: debug 2021-03-09T09:59:10.676+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17066 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:59:15.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:59:15.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:15.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5968 cs=0 l=1 rev1=1 rx=0x7f833c040e50 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:15.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5968 cs=0 l=1 rev1=1 rx=0x7f833c040e50 tx=0x7f833c01af80).stop 2021-03-09T09:59:15.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:15.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:15.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338025120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:15.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:15.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:15.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380168d0 con 0x7f8324003400 2021-03-09T09:59:15.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061d80 con 0x7f83380091b0 2021-03-09T09:59:15.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380153c0 con 0x7f8324005e00 2021-03-09T09:59:15.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:59:15.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:15.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:59:15.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:15.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338025120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:15.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:15.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:15.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:15.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.678+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:15.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.682+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a1c0 con 0x7f83380091b0 2021-03-09T09:59:15.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338025120 secure :-1 s=READY pgs=5977 cs=0 l=1 rev1=1 rx=0x7f833c045580 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:15.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018a30 con 0x7f83380091b0 2021-03-09T09:59:15.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f83380091b0 2021-03-09T09:59:15.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:15.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c045000 con 0x7f83380091b0 2021-03-09T09:59:16.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:59:15 smithi146 bash[13136]: debug 2021-03-09T09:59:15.676+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17091 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:59:20.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.678+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338025120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:59:20.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.678+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338025120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:20.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.678+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338025120 secure :-1 s=READY pgs=5977 cs=0 l=1 rev1=1 rx=0x7f833c045580 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:20.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.678+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338025120 secure :-1 s=READY pgs=5977 cs=0 l=1 rev1=1 rx=0x7f833c045580 tx=0x7f833c01af80).stop 2021-03-09T09:59:20.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 msgr2=0x7f8338025120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:20.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380091b0 0x7f8338025120 unknown :-1 s=CLOSED pgs=5977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:20.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:20.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:20.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:20.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b600 con 0x7f8324003400 2021-03-09T09:59:20.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8338071d10 2021-03-09T09:59:20.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019150 con 0x7f8324005e00 2021-03-09T09:59:20.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:59:20.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:20.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:59:20.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:20.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:20.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:20.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:20.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:20.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:20.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.682+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017550 con 0x7f8338071d10 2021-03-09T09:59:20.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.686+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5987 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:20.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c061ea0 con 0x7f8338071d10 2021-03-09T09:59:20.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0266f0 con 0x7f8338071d10 2021-03-09T09:59:20.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:20.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f8338071d10 2021-03-09T09:59:21.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:59:20 smithi146 bash[13136]: debug 2021-03-09T09:59:20.680+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17116 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:59:25.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.682+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:59:25.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.682+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:25.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5987 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:25.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=5987 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c01af80).stop 2021-03-09T09:59:25.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:25.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=5987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:25.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:25.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:25.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:25.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028660 con 0x7f8324003400 2021-03-09T09:59:25.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380247d0 con 0x7f8338015d30 2021-03-09T09:59:25.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ddb0 con 0x7f8324005e00 2021-03-09T09:59:25.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.682+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:59:25.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.682+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:25.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.682+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:59:25.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:25.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.686+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:25.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.686+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:25.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.686+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:25.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.686+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:25.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.686+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:25.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.686+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b400 con 0x7f8338015d30 2021-03-09T09:59:25.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.690+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380161b0 secure :-1 s=READY pgs=5997 cs=0 l=1 rev1=1 rx=0x7f833c057cf0 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:25.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c061ea0 con 0x7f8338015d30 2021-03-09T09:59:25.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c061ea0 con 0x7f8338015d30 2021-03-09T09:59:25.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:25.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c064930 con 0x7f8338015d30 2021-03-09T09:59:26.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:59:25 smithi146 bash[13136]: debug 2021-03-09T09:59:25.688+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17142 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:59:30.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.686+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:59:30.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.686+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:30.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.686+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380161b0 secure :-1 s=READY pgs=5997 cs=0 l=1 rev1=1 rx=0x7f833c057cf0 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:30.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.686+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380161b0 secure :-1 s=READY pgs=5997 cs=0 l=1 rev1=1 rx=0x7f833c057cf0 tx=0x7f833c01af80).stop 2021-03-09T09:59:30.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380161b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:30.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.686+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380161b0 unknown :-1 s=CLOSED pgs=5997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:30.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.686+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338008870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:30.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.686+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:30.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.686+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:30.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380168d0 con 0x7f8324003400 2021-03-09T09:59:30.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024bf0 con 0x7f8338071d10 2021-03-09T09:59:30.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324005e00 2021-03-09T09:59:30.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.686+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:59:30.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.686+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:30.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.686+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:59:30.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.686+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:30.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.690+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338008870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:30.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.690+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:30.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.690+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:30.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.690+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:30.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.690+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:30.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.690+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014690 con 0x7f8338071d10 2021-03-09T09:59:30.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.690+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338008870 secure :-1 s=READY pgs=6006 cs=0 l=1 rev1=1 rx=0x7f833c079f60 tx=0x7f833c01af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:30.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05c380 con 0x7f8338071d10 2021-03-09T09:59:30.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c048d90 con 0x7f8338071d10 2021-03-09T09:59:30.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:30.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c026380 con 0x7f8338071d10 2021-03-09T09:59:31.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:59:30 smithi146 bash[13136]: debug 2021-03-09T09:59:30.688+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17167 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:59:35.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.690+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338008870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:59:35.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.690+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338008870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:35.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.690+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338008870 secure :-1 s=READY pgs=6006 cs=0 l=1 rev1=1 rx=0x7f833c079f60 tx=0x7f833c01af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:35.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.690+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338008870 secure :-1 s=READY pgs=6006 cs=0 l=1 rev1=1 rx=0x7f833c079f60 tx=0x7f833c01af80).stop 2021-03-09T09:59:35.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338008870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:35.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.690+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338008870 unknown :-1 s=CLOSED pgs=6006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:35.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.690+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:35.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.690+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:59:35.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.690+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:35.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.690+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338013db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:35.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.690+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:35.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f4f0 con 0x7f8324003400 2021-03-09T09:59:35.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028660 con 0x7f8338015d30 2021-03-09T09:59:35.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e160 con 0x7f8324005e00 2021-03-09T09:59:35.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.694+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:59:35.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.694+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:35.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.694+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338013db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:35.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.694+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:35.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.694+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:35.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.694+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:35.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.694+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:35.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.694+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e530 con 0x7f8338015d30 2021-03-09T09:59:35.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.698+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338013db0 secure :-1 s=READY pgs=6015 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:35.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.698+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404b090 con 0x7f8338015d30 2021-03-09T09:59:35.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.698+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344032b80 con 0x7f8338015d30 2021-03-09T09:59:35.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:35.698+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ab60 con 0x7f8338015d30 2021-03-09T09:59:36.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:59:35 smithi146 bash[13136]: debug 2021-03-09T09:59:35.691+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17193 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:59:40.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338013db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:59:40.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338013db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:40.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338013db0 secure :-1 s=READY pgs=6015 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:40.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338013db0 secure :-1 s=READY pgs=6015 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344003f90).stop 2021-03-09T09:59:40.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338013db0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:40.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338013db0 unknown :-1 s=CLOSED pgs=6015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:40.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:40.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:40.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:40.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380168d0 con 0x7f8324003400 2021-03-09T09:59:40.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023ed0 con 0x7f8338071d10 2021-03-09T09:59:40.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d4f0 con 0x7f8324005e00 2021-03-09T09:59:40.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:59:40.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:40.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:59:40.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:40.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:40.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:40.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:40.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:40.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:40.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.694+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f8338071d10 2021-03-09T09:59:40.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.698+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=6020 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c039000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:40.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.698+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c059eb0 con 0x7f8338071d10 2021-03-09T09:59:40.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.698+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c045000 con 0x7f8338071d10 2021-03-09T09:59:40.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:40.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c071b80 con 0x7f8338071d10 2021-03-09T09:59:41.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:59:40 smithi146 bash[13136]: debug 2021-03-09T09:59:40.695+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17218 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:59:45.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:59:45.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:45.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=6020 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c039000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:45.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=6020 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c039000).stop 2021-03-09T09:59:45.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:45.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=6020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:45.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:45.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:45.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:45.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380107d0 con 0x7f8324003400 2021-03-09T09:59:45.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014de0 con 0x7f8338015d30 2021-03-09T09:59:45.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e160 con 0x7f8324005e00 2021-03-09T09:59:45.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:45.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-03-09T09:59:45.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:59:45.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:45.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:45.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:45.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:59:45.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:45.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:45.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380202c0 con 0x7f8338015d30 2021-03-09T09:59:45.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.698+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801dab0 secure :-1 s=READY pgs=6029 cs=0 l=1 rev1=1 rx=0x7f8344073040 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:45.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401e0e0 con 0x7f8338015d30 2021-03-09T09:59:45.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834401e0e0 con 0x7f8338015d30 2021-03-09T09:59:45.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:45.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344007740 con 0x7f8338015d30 2021-03-09T09:59:46.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:59:45 smithi146 bash[13136]: debug 2021-03-09T09:59:45.699+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17244 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:59:50.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.642+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f83380134e0 msgr2=0x7f83380138c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2021-03-09T09:59:50.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.642+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f83380134e0 msgr2=0x7f83380138c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:50.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.642+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f83380134e0 0x7f83380138c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f833405e000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:50.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.642+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f83380134e0 0x7f83380138c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f833405e000).stop 2021-03-09T09:59:50.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.642+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f83380134e0 msgr2=0x7f83380138c0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T09:59:50.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.642+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f83380134e0 msgr2=0x7f83380138c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:50.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.642+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f83380134e0 0x7f83380138c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:50.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.642+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:50.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.642+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f834404aa60 con 0x7f8338071d10 2021-03-09T09:59:50.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.642+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:50.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.642+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f833c018660 tx=0x7f833c039000).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:50.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:59:50.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:50.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801dab0 secure :-1 s=READY pgs=6029 cs=0 l=1 rev1=1 rx=0x7f8344073040 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:50.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801dab0 secure :-1 s=READY pgs=6029 cs=0 l=1 rev1=1 rx=0x7f8344073040 tx=0x7f8344003f90).stop 2021-03-09T09:59:50.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801dab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:50.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801dab0 unknown :-1 s=CLOSED pgs=6029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:50.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:50.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f83380138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:50.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:50.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e160 con 0x7f8324003400 2021-03-09T09:59:50.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018000 con 0x7f83380134e0 2021-03-09T09:59:50.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d4f0 con 0x7f8324005e00 2021-03-09T09:59:50.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:59:50.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:50.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:59:50.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:50.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f83380138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:50.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:50.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:50.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:50.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:50.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020bd0 con 0x7f83380134e0 2021-03-09T09:59:50.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f83380138c0 secure :-1 s=READY pgs=6037 cs=0 l=1 rev1=1 rx=0x7f833806d4e0 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:50.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f83380134e0 2021-03-09T09:59:50.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.706+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344079de0 con 0x7f83380134e0 2021-03-09T09:59:50.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:50.710+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404b650 con 0x7f83380134e0 2021-03-09T09:59:51.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:59:50 smithi146 bash[13136]: debug 2021-03-09T09:59:50.703+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17269 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T09:59:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.702+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f83380138c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:59:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.702+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f83380138c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.702+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f83380138c0 secure :-1 s=READY pgs=6037 cs=0 l=1 rev1=1 rx=0x7f833806d4e0 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.702+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f83380138c0 secure :-1 s=READY pgs=6037 cs=0 l=1 rev1=1 rx=0x7f833806d4e0 tx=0x7f8344003f90).stop 2021-03-09T09:59:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f83380138c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.702+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f83380138c0 unknown :-1 s=CLOSED pgs=6037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.702+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.702+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.702+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380647b0 con 0x7f8324003400 2021-03-09T09:59:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e160 con 0x7f8338015d30 2021-03-09T09:59:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005550 con 0x7f8324005e00 2021-03-09T09:59:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.702+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T09:59:55.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.702+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:55.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.702+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:55.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.706+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T09:59:55.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.706+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:55.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.706+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:55.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.706+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:55.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.706+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:55.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.706+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:55.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.706+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014f20 con 0x7f8338015d30 2021-03-09T09:59:55.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.706+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801c430 secure :-1 s=READY pgs=6046 cs=0 l=1 rev1=1 rx=0x7f834404c5a0 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:55.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.706+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440310e0 con 0x7f8338015d30 2021-03-09T09:59:55.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.706+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834401eca0 con 0x7f8338015d30 2021-03-09T09:59:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T09:59:55.710+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f8338015d30 2021-03-09T09:59:56.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 09:59:55 smithi146 bash[13136]: debug 2021-03-09T09:59:55.707+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17295 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:00:00.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.706+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:00:00.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.706+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:00.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.706+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801c430 secure :-1 s=READY pgs=6046 cs=0 l=1 rev1=1 rx=0x7f834404c5a0 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:00.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.706+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801c430 secure :-1 s=READY pgs=6046 cs=0 l=1 rev1=1 rx=0x7f834404c5a0 tx=0x7f8344003f90).stop 2021-03-09T10:00:00.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.706+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:00.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.706+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801c430 unknown :-1 s=CLOSED pgs=6046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:00.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.706+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:00.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.706+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:00.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.706+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:00.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.706+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008310 con 0x7f8324003400 2021-03-09T10:00:00.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.706+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e160 con 0x7f83380134e0 2021-03-09T10:00:00.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.706+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d60 con 0x7f8324005e00 2021-03-09T10:00:00.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.706+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:00:00.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.706+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:00.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.706+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:00:00.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.706+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:00.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.710+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:00.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.710+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:00.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.710+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:00.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.710+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:00.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.710+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:00.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.710+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017550 con 0x7f83380134e0 2021-03-09T10:00:00.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.710+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6055 cs=0 l=1 rev1=1 rx=0x7f833406e040 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:00.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.714+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401f510 con 0x7f83380134e0 2021-03-09T10:00:00.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.714+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c035820 con 0x7f83380134e0 2021-03-09T10:00:00.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:00.718+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d6e0 con 0x7f83380134e0 2021-03-09T10:00:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:00:00 smithi146 bash[13136]: debug 2021-03-09T10:00:00.707+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17320 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:00:05.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:00:05.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:05.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6055 cs=0 l=1 rev1=1 rx=0x7f833406e040 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:05.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6055 cs=0 l=1 rev1=1 rx=0x7f833406e040 tx=0x7f8334038000).stop 2021-03-09T10:00:05.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:05.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:05.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380106d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:05.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:05.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:05.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a860 con 0x7f8324003400 2021-03-09T10:00:05.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019150 con 0x7f8338015d30 2021-03-09T10:00:05.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017340 con 0x7f8324005e00 2021-03-09T10:00:05.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380106d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:05.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:00:05.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:00:05.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:05.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:05.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:05.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:05.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:05.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:05.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.709+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338060b60 con 0x7f8338015d30 2021-03-09T10:00:05.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.713+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380106d0 secure :-1 s=READY pgs=6063 cs=0 l=1 rev1=1 rx=0x7f8334063750 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:05.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.713+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334063360 con 0x7f8338015d30 2021-03-09T10:00:05.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.713+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f8338015d30 2021-03-09T10:00:05.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:05.717+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340077b0 con 0x7f8338015d30 2021-03-09T10:00:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:00:05 smithi146 bash[13136]: debug 2021-03-09T10:00:05.711+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17346 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:00:10.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380106d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:00:10.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380106d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:10.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380106d0 secure :-1 s=READY pgs=6063 cs=0 l=1 rev1=1 rx=0x7f8334063750 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:10.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380106d0 secure :-1 s=READY pgs=6063 cs=0 l=1 rev1=1 rx=0x7f8334063750 tx=0x7f8334038000).stop 2021-03-09T10:00:10.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380106d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:10.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380106d0 unknown :-1 s=CLOSED pgs=6063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:10.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:10.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:10.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:10.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025620 con 0x7f8324003400 2021-03-09T10:00:10.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380081d0 con 0x7f83380134e0 2021-03-09T10:00:10.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:00:10.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d60 con 0x7f8324005e00 2021-03-09T10:00:10.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:10.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:10.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:00:10.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.713+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:10.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.717+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:10.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.717+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:10.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.717+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:10.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.717+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:10.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.717+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380168d0 con 0x7f83380134e0 2021-03-09T10:00:10.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.717+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6073 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:10.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.717+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334063180 con 0x7f83380134e0 2021-03-09T10:00:10.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.717+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334027e80 con 0x7f83380134e0 2021-03-09T10:00:10.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:10.721+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334064220 con 0x7f83380134e0 2021-03-09T10:00:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:00:10 smithi146 bash[13136]: debug 2021-03-09T10:00:10.715+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17371 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:00:15.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:00:15.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:15.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6073 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:15.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6073 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334038000).stop 2021-03-09T10:00:15.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:15.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:15.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:15.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:15.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:15.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:00:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b400 con 0x7f8324003400 2021-03-09T10:00:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380254e0 con 0x7f8338015d30 2021-03-09T10:00:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ea70 con 0x7f8324005e00 2021-03-09T10:00:15.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:00:15.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:15.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:15.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:15.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.717+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f8338015d30 2021-03-09T10:00:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.721+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 secure :-1 s=READY pgs=6082 cs=0 l=1 rev1=1 rx=0x7f8334031890 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:15.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.721+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334027e80 con 0x7f8338015d30 2021-03-09T10:00:15.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.721+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401a210 con 0x7f8338015d30 2021-03-09T10:00:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:15.725+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334064220 con 0x7f8338015d30 2021-03-09T10:00:16.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:00:15 smithi146 bash[13136]: debug 2021-03-09T10:00:15.719+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17397 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:00:20.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338076600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:00:20.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338076600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:20.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 secure :-1 s=READY pgs=6082 cs=0 l=1 rev1=1 rx=0x7f8334031890 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:20.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 secure :-1 s=READY pgs=6082 cs=0 l=1 rev1=1 rx=0x7f8334031890 tx=0x7f8334038000).stop 2021-03-09T10:00:20.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338076600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:20.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 unknown :-1 s=CLOSED pgs=6082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:20.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:20.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:20.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:20.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:00:20.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:20.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:20.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e160 con 0x7f8324003400 2021-03-09T10:00:20.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008310 con 0x7f83380134e0 2021-03-09T10:00:20.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d60 con 0x7f8324005e00 2021-03-09T10:00:20.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:00:20.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:20.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:20.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:20.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:20.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:20.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.721+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017550 con 0x7f83380134e0 2021-03-09T10:00:20.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.725+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6092 cs=0 l=1 rev1=1 rx=0x7f833406d900 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:20.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.725+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833406d180 con 0x7f83380134e0 2021-03-09T10:00:20.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.729+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833400dda0 con 0x7f83380134e0 2021-03-09T10:00:20.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:20.729+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340164b0 con 0x7f83380134e0 2021-03-09T10:00:21.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:00:20 smithi146 bash[13136]: debug 2021-03-09T10:00:20.723+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17423 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:00:25.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:00:25.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:25.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6092 cs=0 l=1 rev1=1 rx=0x7f833406d900 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:25.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6092 cs=0 l=1 rev1=1 rx=0x7f833406d900 tx=0x7f8334038000).stop 2021-03-09T10:00:25.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:25.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:25.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:25.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:25.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:25.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:00:25.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:25.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b400 con 0x7f8324003400 2021-03-09T10:00:25.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025620 con 0x7f8338015d30 2021-03-09T10:00:25.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324005e00 2021-03-09T10:00:25.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:25.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:00:25.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:25.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:25.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:25.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:25.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:25.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b120 con 0x7f8338015d30 2021-03-09T10:00:25.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.725+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801cc40 secure :-1 s=READY pgs=6098 cs=0 l=1 rev1=1 rx=0x7f833c028000 tx=0x7f8344052000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:25.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.733+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0416c0 con 0x7f8338015d30 2021-03-09T10:00:25.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.733+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404aa60 con 0x7f8338015d30 2021-03-09T10:00:25.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:25.737+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338015d30 2021-03-09T10:00:26.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:00:25 smithi146 bash[13136]: debug 2021-03-09T10:00:25.723+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17449 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:00:30.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:00:30.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:30.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801cc40 secure :-1 s=READY pgs=6098 cs=0 l=1 rev1=1 rx=0x7f833c028000 tx=0x7f8344052000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:30.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801cc40 secure :-1 s=READY pgs=6098 cs=0 l=1 rev1=1 rx=0x7f833c028000 tx=0x7f8344052000).stop 2021-03-09T10:00:30.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801cc40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:30.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801cc40 unknown :-1 s=CLOSED pgs=6098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:30.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:30.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:30.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:30.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380081d0 con 0x7f8324003400 2021-03-09T10:00:30.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ea70 con 0x7f83380134e0 2021-03-09T10:00:30.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324005e00 2021-03-09T10:00:30.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:00:30.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:30.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:00:30.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:30.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.725+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:30.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.729+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:30.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.729+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:30.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.729+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:30.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.729+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:30.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.729+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801f970 con 0x7f83380134e0 2021-03-09T10:00:30.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.729+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6107 cs=0 l=1 rev1=1 rx=0x7f833c0224e0 tx=0x7f8344052000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:30.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.729+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c026b20 con 0x7f83380134e0 2021-03-09T10:00:30.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.729+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c026d00 con 0x7f83380134e0 2021-03-09T10:00:30.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:30.733+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05b800 con 0x7f83380134e0 2021-03-09T10:00:31.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:00:30 smithi146 bash[13136]: debug 2021-03-09T10:00:30.727+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17474 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:00:35.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:00:35.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:35.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6107 cs=0 l=1 rev1=1 rx=0x7f833c0224e0 tx=0x7f8344052000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:35.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6107 cs=0 l=1 rev1=1 rx=0x7f833c0224e0 tx=0x7f8344052000).stop 2021-03-09T10:00:35.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:35.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:35.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:35.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:35.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:35.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d60 con 0x7f8324003400 2021-03-09T10:00:35.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e120 con 0x7f8338015d30 2021-03-09T10:00:35.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a8b0 con 0x7f8324005e00 2021-03-09T10:00:35.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:00:35.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:35.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:00:35.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:35.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.729+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:35.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.733+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:35.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.733+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:35.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.733+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:35.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.733+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:35.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.733+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f8338015d30 2021-03-09T10:00:35.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.733+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801d430 secure :-1 s=READY pgs=6115 cs=0 l=1 rev1=1 rx=0x7f833404b8e0 tx=0x7f833406d870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:35.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.733+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334052520 con 0x7f8338015d30 2021-03-09T10:00:35.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.733+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340314f0 con 0x7f8338015d30 2021-03-09T10:00:35.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:35.737+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334057b50 con 0x7f8338015d30 2021-03-09T10:00:36.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:00:35 smithi146 bash[13136]: debug 2021-03-09T10:00:35.731+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17495 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:00:40.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:00:40.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:40.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801d430 secure :-1 s=READY pgs=6115 cs=0 l=1 rev1=1 rx=0x7f833404b8e0 tx=0x7f833406d870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:40.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801d430 secure :-1 s=READY pgs=6115 cs=0 l=1 rev1=1 rx=0x7f833404b8e0 tx=0x7f833406d870).stop 2021-03-09T10:00:40.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801d430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:40.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801d430 unknown :-1 s=CLOSED pgs=6115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:40.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:40.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:40.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:40.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b400 con 0x7f8324003400 2021-03-09T10:00:40.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ea70 con 0x7f83380134e0 2021-03-09T10:00:40.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324005e00 2021-03-09T10:00:40.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:00:40.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:40.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:00:40.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:40.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.733+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:40.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.737+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:40.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.737+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:40.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.737+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:40.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.737+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:40.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.737+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f83380134e0 2021-03-09T10:00:40.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.737+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6125 cs=0 l=1 rev1=1 rx=0x7f8334018e70 tx=0x7f833406d870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:40.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.737+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334024000 con 0x7f83380134e0 2021-03-09T10:00:40.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.737+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334024640 con 0x7f83380134e0 2021-03-09T10:00:40.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:40.741+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833406d490 con 0x7f83380134e0 2021-03-09T10:00:41.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:00:40 smithi146 bash[13136]: debug 2021-03-09T10:00:40.735+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17521 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:00:45.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.733+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:00:45.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.733+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:45.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.733+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6125 cs=0 l=1 rev1=1 rx=0x7f8334018e70 tx=0x7f833406d870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:45.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.733+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6125 cs=0 l=1 rev1=1 rx=0x7f8334018e70 tx=0x7f833406d870).stop 2021-03-09T10:00:45.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:45.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:45.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:45.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:45.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:00:45.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:45.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:45.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:45.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380639a0 con 0x7f8324003400 2021-03-09T10:00:45.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062090 con 0x7f8338015d30 2021-03-09T10:00:45.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d60 con 0x7f8324005e00 2021-03-09T10:00:45.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:00:45.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:45.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:45.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:45.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:45.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:45.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.737+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f8338015d30 2021-03-09T10:00:45.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.741+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069bb0 secure :-1 s=READY pgs=6135 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f833406d870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:45.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.741+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833403a3d0 con 0x7f8338015d30 2021-03-09T10:00:45.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.741+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403a810 con 0x7f8338015d30 2021-03-09T10:00:45.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:45.745+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334024000 con 0x7f8338015d30 2021-03-09T10:00:46.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:00:45 smithi146 bash[13136]: debug 2021-03-09T10:00:45.738+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17549 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:00:50.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338069bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:00:50.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338069bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:50.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069bb0 secure :-1 s=READY pgs=6135 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f833406d870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:50.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069bb0 secure :-1 s=READY pgs=6135 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f833406d870).stop 2021-03-09T10:00:50.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338069bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:50.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069bb0 unknown :-1 s=CLOSED pgs=6135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:50.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:50.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007930 con 0x7f8324003400 2021-03-09T10:00:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f4f0 con 0x7f83380134e0 2021-03-09T10:00:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060c00 con 0x7f8324005e00 2021-03-09T10:00:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:00:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:00:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ed40 con 0x7f83380134e0 2021-03-09T10:00:50.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.741+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6141 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:50.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.745+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f83380134e0 2021-03-09T10:00:50.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.745+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404aa60 con 0x7f83380134e0 2021-03-09T10:00:50.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:50.745+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440791e0 con 0x7f83380134e0 2021-03-09T10:00:51.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:00:50 smithi146 bash[13136]: debug 2021-03-09T10:00:50.742+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17567 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:00:55.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.741+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:00:55.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.741+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.741+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6141 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.741+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6141 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344003f90).stop 2021-03-09T10:00:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.741+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.741+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.741+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:55.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.741+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:55.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.741+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:55.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.741+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ba00 con 0x7f8324003400 2021-03-09T10:00:55.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.741+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8338015d30 2021-03-09T10:00:55.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.741+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064050 con 0x7f8324005e00 2021-03-09T10:00:55.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.741+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:00:55.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.741+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:00:55.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.741+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:55.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.741+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:55.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.745+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:55.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.745+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:55.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.745+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:55.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.745+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:55.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.745+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:55.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.745+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e310 con 0x7f8338015d30 2021-03-09T10:00:55.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.749+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069bb0 secure :-1 s=READY pgs=6152 cs=0 l=1 rev1=1 rx=0x7f8334022040 tx=0x7f833406d870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:55.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.749+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404d610 con 0x7f8338015d30 2021-03-09T10:00:55.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.749+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404d610 con 0x7f8338015d30 2021-03-09T10:00:55.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:00:55.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334027250 con 0x7f8338015d30 2021-03-09T10:00:56.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:00:55 smithi146 bash[13136]: debug 2021-03-09T10:00:55.742+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17600 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:01:00.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338069bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:01:00.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338069bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:00.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069bb0 secure :-1 s=READY pgs=6152 cs=0 l=1 rev1=1 rx=0x7f8334022040 tx=0x7f833406d870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:00.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069bb0 secure :-1 s=READY pgs=6152 cs=0 l=1 rev1=1 rx=0x7f8334022040 tx=0x7f833406d870).stop 2021-03-09T10:01:00.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338069bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:00.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069bb0 unknown :-1 s=CLOSED pgs=6152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:00.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:00.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:00.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:00.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d60 con 0x7f8324003400 2021-03-09T10:01:00.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dfb0 con 0x7f83380134e0 2021-03-09T10:01:00.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324005e00 2021-03-09T10:01:00.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:01:00.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:00.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:01:00.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:00.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:00.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:00.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:00.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f833800c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:00.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f833800c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:00.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f83380134e0 2021-03-09T10:01:00.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6159 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:00.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f83380134e0 2021-03-09T10:01:00.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.748+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405e270 con 0x7f83380134e0 2021-03-09T10:01:00.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:00.752+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344049710 con 0x7f83380134e0 2021-03-09T10:01:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:01:00 smithi146 bash[13136]: debug 2021-03-09T10:01:00.746+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17626 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:01:05.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:01:05.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:05.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6159 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:05.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6159 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004520).stop 2021-03-09T10:01:05.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:05.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:05.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:05.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:05.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:05.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ba00 con 0x7f8324003400 2021-03-09T10:01:05.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064050 con 0x7f8338015d30 2021-03-09T10:01:05.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324005e00 2021-03-09T10:01:05.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:01:05.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:05.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:01:05.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:05.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.748+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:05.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.752+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:05.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.752+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:05.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.752+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:05.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.752+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:05.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.752+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f8338015d30 2021-03-09T10:01:05.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.752+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c720 secure :-1 s=READY pgs=6170 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:05.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.752+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405bc30 con 0x7f8338015d30 2021-03-09T10:01:05.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.752+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405e270 con 0x7f8338015d30 2021-03-09T10:01:05.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:05.756+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440171c0 con 0x7f8338015d30 2021-03-09T10:01:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:01:05 smithi146 bash[13136]: debug 2021-03-09T10:01:05.750+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17647 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:01:10.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.748+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:01:10.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.748+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:10.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.748+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c720 secure :-1 s=READY pgs=6170 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:10.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.748+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c720 secure :-1 s=READY pgs=6170 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344004520).stop 2021-03-09T10:01:10.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800c720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:10.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c720 unknown :-1 s=CLOSED pgs=6170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:10.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:10.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:10.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:10.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T10:01:10.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011320 con 0x7f83380134e0 2021-03-09T10:01:10.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d60 con 0x7f8324005e00 2021-03-09T10:01:10.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:01:10.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:10.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:01:10.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:10.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:10.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:10.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:10.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:10.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:10.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.752+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f83380134e0 2021-03-09T10:01:10.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.756+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6179 cs=0 l=1 rev1=1 rx=0x7f8344004690 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:10.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.756+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344054d80 con 0x7f83380134e0 2021-03-09T10:01:10.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.756+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344054d80 con 0x7f83380134e0 2021-03-09T10:01:10.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:10.760+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344078190 con 0x7f83380134e0 2021-03-09T10:01:11.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:01:10 smithi146 bash[13136]: debug 2021-03-09T10:01:10.754+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17677 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:01:15.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:01:15.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:15.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6179 cs=0 l=1 rev1=1 rx=0x7f8344004690 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:15.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6179 cs=0 l=1 rev1=1 rx=0x7f8344004690 tx=0x7f8344004520).stop 2021-03-09T10:01:15.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:15.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:15.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:15.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:15.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:15.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:01:15.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:15.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:01:15.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:15.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:15.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324003400 2021-03-09T10:01:15.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b400 con 0x7f8338015d30 2021-03-09T10:01:15.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ba00 con 0x7f8324005e00 2021-03-09T10:01:15.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:15.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:15.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:15.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:15.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f8338015d30 2021-03-09T10:01:15.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c560 secure :-1 s=READY pgs=6185 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:15.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403cc10 con 0x7f8338015d30 2021-03-09T10:01:15.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.756+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344048dd0 con 0x7f8338015d30 2021-03-09T10:01:15.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:15.760+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344054be0 con 0x7f8338015d30 2021-03-09T10:01:16.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:01:15 smithi146 bash[13136]: debug 2021-03-09T10:01:15.758+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17701 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:01:20.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:01:20.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:20.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c560 secure :-1 s=READY pgs=6185 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:20.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c560 secure :-1 s=READY pgs=6185 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344004520).stop 2021-03-09T10:01:20.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800c560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:20.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c560 unknown :-1 s=CLOSED pgs=6185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:20.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:20.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:20.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:20.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011320 con 0x7f8324003400 2021-03-09T10:01:20.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d60 con 0x7f83380134e0 2021-03-09T10:01:20.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064050 con 0x7f8324005e00 2021-03-09T10:01:20.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:01:20.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:20.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:01:20.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:20.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:20.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:20.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:20.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:20.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:20.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f83380134e0 2021-03-09T10:01:20.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6193 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:20.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019cc0 con 0x7f83380134e0 2021-03-09T10:01:20.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.760+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405e270 con 0x7f83380134e0 2021-03-09T10:01:20.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:20.764+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834401fd90 con 0x7f83380134e0 2021-03-09T10:01:21.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:01:20 smithi146 bash[13136]: debug 2021-03-09T10:01:20.758+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17728 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:01:25.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:01:25.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:25.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6193 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:25.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6193 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004520).stop 2021-03-09T10:01:25.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:25.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:25.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380686d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:25.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:25.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:25.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324003400 2021-03-09T10:01:25.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380076f0 con 0x7f8338015d30 2021-03-09T10:01:25.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060b60 con 0x7f8324005e00 2021-03-09T10:01:25.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:01:25.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:25.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:01:25.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:25.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380686d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:25.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:25.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:25.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:25.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:25.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f8338015d30 2021-03-09T10:01:25.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380686d0 secure :-1 s=READY pgs=6202 cs=0 l=1 rev1=1 rx=0x7f8334033420 tx=0x7f833406d870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:25.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833405c9c0 con 0x7f8338015d30 2021-03-09T10:01:25.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.764+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833405cb20 con 0x7f8338015d30 2021-03-09T10:01:25.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:25.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340077b0 con 0x7f8338015d30 2021-03-09T10:01:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:01:25 smithi146 bash[13136]: debug 2021-03-09T10:01:25.762+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17752 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:01:30.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380686d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:01:30.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380686d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:30.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380686d0 secure :-1 s=READY pgs=6202 cs=0 l=1 rev1=1 rx=0x7f8334033420 tx=0x7f833406d870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:30.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380686d0 secure :-1 s=READY pgs=6202 cs=0 l=1 rev1=1 rx=0x7f8334033420 tx=0x7f833406d870).stop 2021-03-09T10:01:30.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380686d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:30.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380686d0 unknown :-1 s=CLOSED pgs=6202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:30.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:30.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:30.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:30.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c360 con 0x7f8324003400 2021-03-09T10:01:30.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011320 con 0x7f83380134e0 2021-03-09T10:01:30.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324005e00 2021-03-09T10:01:30.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:01:30.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:30.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:01:30.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:30.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:30.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:30.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:30.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:30.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:30.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064050 con 0x7f83380134e0 2021-03-09T10:01:30.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.768+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6213 cs=0 l=1 rev1=1 rx=0x7f8338009e70 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:30.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019cc0 con 0x7f83380134e0 2021-03-09T10:01:30.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f83380134e0 2021-03-09T10:01:30.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:30.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405ece0 con 0x7f83380134e0 2021-03-09T10:01:31.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:01:30 smithi146 bash[13136]: debug 2021-03-09T10:01:30.766+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17779 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:01:35.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:01:35.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:35.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6213 cs=0 l=1 rev1=1 rx=0x7f8338009e70 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:35.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6213 cs=0 l=1 rev1=1 rx=0x7f8338009e70 tx=0x7f8344004520).stop 2021-03-09T10:01:35.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:35.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:35.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:35.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:35.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:35.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324003400 2021-03-09T10:01:35.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028400 con 0x7f8338015d30 2021-03-09T10:01:35.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020b50 con 0x7f8324005e00 2021-03-09T10:01:35.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:01:35.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:01:35.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:35.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:35.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:35.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:35.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:35.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:35.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:35.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f8338015d30 2021-03-09T10:01:35.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802c7e0 secure :-1 s=READY pgs=6219 cs=0 l=1 rev1=1 rx=0x7f833c01bf50 tx=0x7f8344052000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:35.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c054310 con 0x7f8338015d30 2021-03-09T10:01:35.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05b410 con 0x7f8338015d30 2021-03-09T10:01:35.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:35.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338015d30 2021-03-09T10:01:36.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:01:35 smithi146 bash[13136]: debug 2021-03-09T10:01:35.770+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17803 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:01:40.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.767+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:01:40.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.767+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:40.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.767+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802c7e0 secure :-1 s=READY pgs=6219 cs=0 l=1 rev1=1 rx=0x7f833c01bf50 tx=0x7f8344052000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:40.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.767+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802c7e0 secure :-1 s=READY pgs=6219 cs=0 l=1 rev1=1 rx=0x7f833c01bf50 tx=0x7f8344052000).stop 2021-03-09T10:01:40.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.767+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802c7e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:40.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.767+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802c7e0 unknown :-1 s=CLOSED pgs=6219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:40.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.767+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:40.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.771+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:40.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.771+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.771+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380647f0 con 0x7f8324003400 2021-03-09T10:01:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.771+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f83380134e0 2021-03-09T10:01:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.771+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020b50 con 0x7f8324005e00 2021-03-09T10:01:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.771+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:01:40.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.771+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:01:40.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.771+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:40.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.771+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:40.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.771+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:40.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.775+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:40.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.775+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:40.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.775+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:40.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.775+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:40.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.775+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f83380134e0 2021-03-09T10:01:40.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.779+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6229 cs=0 l=1 rev1=1 rx=0x7f8334024a40 tx=0x7f833406d870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:40.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.779+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400bab0 con 0x7f83380134e0 2021-03-09T10:01:40.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.779+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334016bd0 con 0x7f83380134e0 2021-03-09T10:01:40.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:40.783+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334016bd0 con 0x7f83380134e0 2021-03-09T10:01:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:01:40 smithi146 bash[13136]: debug 2021-03-09T10:01:40.774+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17830 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:01:45.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.775+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:01:45.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.775+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:45.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.775+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6229 cs=0 l=1 rev1=1 rx=0x7f8334024a40 tx=0x7f833406d870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:45.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.775+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6229 cs=0 l=1 rev1=1 rx=0x7f8334024a40 tx=0x7f833406d870).stop 2021-03-09T10:01:45.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.775+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:45.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.775+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:45.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.775+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:45.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.775+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:45.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.775+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:45.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.775+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:01:45.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.775+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:45.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.779+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:01:45.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.779+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020320 con 0x7f8324003400 2021-03-09T10:01:45.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.779+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8338015d30 2021-03-09T10:01:45.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.779+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008420 con 0x7f8324005e00 2021-03-09T10:01:45.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.779+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:45.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.779+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:45.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.779+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:45.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.779+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:45.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.779+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:45.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.779+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:45.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.779+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f8338015d30 2021-03-09T10:01:45.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.779+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c560 secure :-1 s=READY pgs=6237 cs=0 l=1 rev1=1 rx=0x7f833c0224e0 tx=0x7f8344052000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:45.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.783+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c007980 con 0x7f8338015d30 2021-03-09T10:01:45.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.783+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f8338015d30 2021-03-09T10:01:45.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:45.783+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007380 con 0x7f8338015d30 2021-03-09T10:01:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:01:45 smithi146 bash[13136]: debug 2021-03-09T10:01:45.778+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17854 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:01:50.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.779+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:01:50.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.779+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:50.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.779+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c560 secure :-1 s=READY pgs=6237 cs=0 l=1 rev1=1 rx=0x7f833c0224e0 tx=0x7f8344052000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:50.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.779+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c560 secure :-1 s=READY pgs=6237 cs=0 l=1 rev1=1 rx=0x7f833c0224e0 tx=0x7f8344052000).stop 2021-03-09T10:01:50.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.779+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800c560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:50.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.779+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800c560 unknown :-1 s=CLOSED pgs=6237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:50.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.779+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:50.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.779+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:50.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.783+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:50.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.783+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028400 con 0x7f8324003400 2021-03-09T10:01:50.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.783+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008420 con 0x7f83380134e0 2021-03-09T10:01:50.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.783+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021780 con 0x7f8324005e00 2021-03-09T10:01:50.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.783+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:01:50.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.783+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:01:50.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.783+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:50.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.783+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:50.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.783+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:50.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.783+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:50.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.783+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:50.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.783+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:50.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.783+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:50.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.783+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010f20 con 0x7f83380134e0 2021-03-09T10:01:50.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.783+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6248 cs=0 l=1 rev1=1 rx=0x7f8334004230 tx=0x7f833406d870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:50.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.787+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404d170 con 0x7f83380134e0 2021-03-09T10:01:50.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.787+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404d3a0 con 0x7f83380134e0 2021-03-09T10:01:50.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:50.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403a1f0 con 0x7f83380134e0 2021-03-09T10:01:51.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:01:50 smithi146 bash[13136]: debug 2021-03-09T10:01:50.778+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17879 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:01:55.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:01:55.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:55.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6248 cs=0 l=1 rev1=1 rx=0x7f8334004230 tx=0x7f833406d870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:55.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6248 cs=0 l=1 rev1=1 rx=0x7f8334004230 tx=0x7f833406d870).stop 2021-03-09T10:01:55.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324003400 2021-03-09T10:01:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020320 con 0x7f8338015d30 2021-03-09T10:01:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008420 con 0x7f8324005e00 2021-03-09T10:01:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:01:55.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:01:55.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:55.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:55.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.783+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:55.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.787+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:55.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.787+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:55.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.787+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:55.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.787+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:55.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.787+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338023880 con 0x7f8338015d30 2021-03-09T10:01:55.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.787+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802dbb0 secure :-1 s=READY pgs=6255 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f8344052000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:55.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.787+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f8338015d30 2021-03-09T10:01:55.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.787+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f8338015d30 2021-03-09T10:01:55.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:01:55.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338015d30 2021-03-09T10:01:56.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:01:55 smithi146 bash[13136]: debug 2021-03-09T10:01:55.781+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17905 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:02:00.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.787+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:02:00.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.787+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:00.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.787+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802dbb0 secure :-1 s=READY pgs=6255 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f8344052000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:00.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.787+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802dbb0 secure :-1 s=READY pgs=6255 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f8344052000).stop 2021-03-09T10:02:00.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.787+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802dbb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:00.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.787+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802dbb0 unknown :-1 s=CLOSED pgs=6255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:00.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.787+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:00.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.787+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:02:00.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.787+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:00.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.787+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:00.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.787+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:00.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.787+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bb80 con 0x7f8324003400 2021-03-09T10:02:00.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.787+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008420 con 0x7f83380134e0 2021-03-09T10:02:00.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.787+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021780 con 0x7f8324005e00 2021-03-09T10:02:00.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.787+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:02:00.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.787+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:00.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:00.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.791+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:00.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:00.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.791+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:00.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:00.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.791+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b120 con 0x7f83380134e0 2021-03-09T10:02:00.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6264 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c07c040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:00.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0412e0 con 0x7f83380134e0 2021-03-09T10:02:00.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f83380134e0 2021-03-09T10:02:00.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:00.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c041d80 con 0x7f83380134e0 2021-03-09T10:02:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:02:00 smithi146 bash[13136]: debug 2021-03-09T10:02:00.785+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17932 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:02:05.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:02:05.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:05.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6264 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c07c040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:05.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6264 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c07c040).stop 2021-03-09T10:02:05.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:05.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:05.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380184b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:05.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:05.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:05.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020320 con 0x7f8324003400 2021-03-09T10:02:05.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8338015d30 2021-03-09T10:02:05.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008420 con 0x7f8324005e00 2021-03-09T10:02:05.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:02:05.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:05.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:02:05.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:05.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380184b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:05.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:05.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:05.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:05.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:05.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.791+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801d8d0 con 0x7f8338015d30 2021-03-09T10:02:05.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.795+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380184b0 secure :-1 s=READY pgs=6271 cs=0 l=1 rev1=1 rx=0x7f8334009b10 tx=0x7f833406d870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:05.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334016d90 con 0x7f8338015d30 2021-03-09T10:02:05.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334024ac0 con 0x7f8338015d30 2021-03-09T10:02:05.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:05.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340077b0 con 0x7f8338015d30 2021-03-09T10:02:06.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:02:05 smithi146 bash[13136]: debug 2021-03-09T10:02:05.789+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17956 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:02:10.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380184b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:02:10.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380184b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:10.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380184b0 secure :-1 s=READY pgs=6271 cs=0 l=1 rev1=1 rx=0x7f8334009b10 tx=0x7f833406d870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:10.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380184b0 secure :-1 s=READY pgs=6271 cs=0 l=1 rev1=1 rx=0x7f8334009b10 tx=0x7f833406d870).stop 2021-03-09T10:02:10.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380184b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:10.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380184b0 unknown :-1 s=CLOSED pgs=6271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:10.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:10.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:10.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:10.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028400 con 0x7f8324003400 2021-03-09T10:02:10.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f83380134e0 2021-03-09T10:02:10.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008420 con 0x7f8324005e00 2021-03-09T10:02:10.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:02:10.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:10.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:02:10.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:10.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.795+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:10.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.799+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:10.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.799+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:10.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.799+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:10.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.799+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:10.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.799+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f83380134e0 2021-03-09T10:02:10.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.799+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6282 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:10.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f83380134e0 2021-03-09T10:02:10.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019cc0 con 0x7f83380134e0 2021-03-09T10:02:10.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:10.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440791e0 con 0x7f83380134e0 2021-03-09T10:02:11.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:02:10 smithi146 bash[13136]: debug 2021-03-09T10:02:10.793+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 17983 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:02:15.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.795+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:02:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.795+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.795+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6282 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.795+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6282 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004520).stop 2021-03-09T10:02:15.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.795+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:15.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.795+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:15.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.795+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:15.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380184b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:15.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:15.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324003400 2021-03-09T10:02:15.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020320 con 0x7f8338015d30 2021-03-09T10:02:15.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008420 con 0x7f8324005e00 2021-03-09T10:02:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.799+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:02:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.799+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:02:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.799+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.799+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:15.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.799+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380184b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:15.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.799+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.799+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.799+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.799+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:15.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.799+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f8338015d30 2021-03-09T10:02:15.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.803+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380184b0 secure :-1 s=READY pgs=6292 cs=0 l=1 rev1=1 rx=0x7f8344078b80 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:15.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440787e0 con 0x7f8338015d30 2021-03-09T10:02:15.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344078610 con 0x7f8338015d30 2021-03-09T10:02:15.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:15.807+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440173d0 con 0x7f8338015d30 2021-03-09T10:02:16.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:02:15 smithi146 bash[13136]: debug 2021-03-09T10:02:15.797+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18007 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:02:20.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.799+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380184b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:02:20.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.799+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380184b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:20.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.799+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380184b0 secure :-1 s=READY pgs=6292 cs=0 l=1 rev1=1 rx=0x7f8344078b80 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:20.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.799+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380184b0 secure :-1 s=READY pgs=6292 cs=0 l=1 rev1=1 rx=0x7f8344078b80 tx=0x7f8344004520).stop 2021-03-09T10:02:20.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380184b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:20.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380184b0 unknown :-1 s=CLOSED pgs=6292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:20.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:20.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:20.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.799+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:20.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.799+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324003400 2021-03-09T10:02:20.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f83380134e0 2021-03-09T10:02:20.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008420 con 0x7f8324005e00 2021-03-09T10:02:20.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.803+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:02:20.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.803+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:20.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.803+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:02:20.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.803+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:20.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:20.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.803+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:20.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:20.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.803+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:20.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:20.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.803+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010f20 con 0x7f83380134e0 2021-03-09T10:02:20.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.803+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6297 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c07c040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:20.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.803+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c057900 con 0x7f83380134e0 2021-03-09T10:02:20.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.807+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0413b0 con 0x7f83380134e0 2021-03-09T10:02:20.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:20.807+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f83380134e0 2021-03-09T10:02:21.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:02:20 smithi146 bash[13136]: debug 2021-03-09T10:02:20.801+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18034 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:02:25.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:02:25.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:25.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6297 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c07c040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:25.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6297 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c07c040).stop 2021-03-09T10:02:25.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:25.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:25.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:25.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:25.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:25.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025a70 con 0x7f8324003400 2021-03-09T10:02:25.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008420 con 0x7f8338015d30 2021-03-09T10:02:25.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006930 con 0x7f8324005e00 2021-03-09T10:02:25.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:02:25.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:25.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:02:25.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:25.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.802+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:25.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.806+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:25.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.806+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:25.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.806+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:25.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.806+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:25.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.806+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801d8d0 con 0x7f8338015d30 2021-03-09T10:02:25.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.806+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 secure :-1 s=READY pgs=6306 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:25.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.806+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405bec0 con 0x7f8338015d30 2021-03-09T10:02:25.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.806+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834406ce90 con 0x7f8338015d30 2021-03-09T10:02:25.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:25.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440791e0 con 0x7f8338015d30 2021-03-09T10:02:26.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:02:25 smithi146 bash[13136]: debug 2021-03-09T10:02:25.801+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18058 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:02:30.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:02:30.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:30.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 secure :-1 s=READY pgs=6306 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:30.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 secure :-1 s=READY pgs=6306 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344004520).stop 2021-03-09T10:02:30.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338063080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:30.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 unknown :-1 s=CLOSED pgs=6306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:30.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:30.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:30.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:30.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a30 con 0x7f8324003400 2021-03-09T10:02:30.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380082a0 con 0x7f83380134e0 2021-03-09T10:02:30.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d790 con 0x7f8324005e00 2021-03-09T10:02:30.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:02:30.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:30.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:02:30.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:30.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.806+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:30.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.810+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:30.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.810+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:30.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.810+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:30.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.810+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:30.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.810+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f83380134e0 2021-03-09T10:02:30.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.810+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6315 cs=0 l=1 rev1=1 rx=0x7f833400d700 tx=0x7f833406d870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:30.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833405c630 con 0x7f83380134e0 2021-03-09T10:02:30.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833405c790 con 0x7f83380134e0 2021-03-09T10:02:30.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:30.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833406d1f0 con 0x7f83380134e0 2021-03-09T10:02:31.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:02:30 smithi146 bash[13136]: debug 2021-03-09T10:02:30.805+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18084 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:02:35.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:02:35.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:35.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6315 cs=0 l=1 rev1=1 rx=0x7f833400d700 tx=0x7f833406d870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:35.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6315 cs=0 l=1 rev1=1 rx=0x7f833400d700 tx=0x7f833406d870).stop 2021-03-09T10:02:35.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:35.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:35.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:35.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:35.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:35.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008420 con 0x7f8324003400 2021-03-09T10:02:35.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8338015d30 2021-03-09T10:02:35.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068fb0 con 0x7f8324005e00 2021-03-09T10:02:35.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:02:35.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:02:35.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:35.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:35.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.810+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:35.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.814+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:35.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.814+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:35.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.814+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:35.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.814+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:35.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.814+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338060d10 con 0x7f8338015d30 2021-03-09T10:02:35.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.814+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 secure :-1 s=READY pgs=6323 cs=0 l=1 rev1=1 rx=0x7f834402c890 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:35.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402c2c0 con 0x7f8338015d30 2021-03-09T10:02:35.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834406ce90 con 0x7f8338015d30 2021-03-09T10:02:35.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:35.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440791e0 con 0x7f8338015d30 2021-03-09T10:02:36.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:02:35 smithi146 bash[13136]: debug 2021-03-09T10:02:35.809+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18109 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:02:40.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:02:40.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:40.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 secure :-1 s=READY pgs=6323 cs=0 l=1 rev1=1 rx=0x7f834402c890 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:40.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 secure :-1 s=READY pgs=6323 cs=0 l=1 rev1=1 rx=0x7f834402c890 tx=0x7f8344004520).stop 2021-03-09T10:02:40.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338063080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:40.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 unknown :-1 s=CLOSED pgs=6323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:40.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:40.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:02:40.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:40.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:40.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:40.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324003400 2021-03-09T10:02:40.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008420 con 0x7f83380134e0 2021-03-09T10:02:40.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068fb0 con 0x7f8324005e00 2021-03-09T10:02:40.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:02:40.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:40.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.814+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:40.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.818+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:40.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:40.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.818+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:40.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:40.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.818+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f83380134e0 2021-03-09T10:02:40.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6335 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:40.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403ba10 con 0x7f83380134e0 2021-03-09T10:02:40.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834406ce90 con 0x7f83380134e0 2021-03-09T10:02:40.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:40.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440105a0 con 0x7f83380134e0 2021-03-09T10:02:41.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:02:40 smithi146 bash[13136]: debug 2021-03-09T10:02:40.813+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18135 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:02:45.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.814+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:02:45.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.814+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:45.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.814+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6335 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:45.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.814+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6335 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344004520).stop 2021-03-09T10:02:45.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.814+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:45.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.814+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:45.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.814+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:45.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:02:45.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:45.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:45.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:45.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021780 con 0x7f8324003400 2021-03-09T10:02:45.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8338015d30 2021-03-09T10:02:45.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802b4d0 con 0x7f8324005e00 2021-03-09T10:02:45.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:02:45.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:45.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:45.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:45.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:45.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:45.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:45.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010f20 con 0x7f8338015d30 2021-03-09T10:02:45.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 secure :-1 s=READY pgs=6341 cs=0 l=1 rev1=1 rx=0x7f8344003ac0 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:45.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.818+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402b400 con 0x7f8338015d30 2021-03-09T10:02:45.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834406ce90 con 0x7f8338015d30 2021-03-09T10:02:45.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:45.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344067320 con 0x7f8338015d30 2021-03-09T10:02:46.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:02:45 smithi146 bash[13136]: debug 2021-03-09T10:02:45.817+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18160 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:02:50.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:02:50.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:50.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 secure :-1 s=READY pgs=6341 cs=0 l=1 rev1=1 rx=0x7f8344003ac0 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:50.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 secure :-1 s=READY pgs=6341 cs=0 l=1 rev1=1 rx=0x7f8344003ac0 tx=0x7f8344004520).stop 2021-03-09T10:02:50.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338063080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:50.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063080 unknown :-1 s=CLOSED pgs=6341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:50.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:50.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:50.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:50.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324003400 2021-03-09T10:02:50.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f83380134e0 2021-03-09T10:02:50.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006930 con 0x7f8324005e00 2021-03-09T10:02:50.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:02:50.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:50.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:02:50.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:50.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:50.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:50.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:50.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:50.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:50.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065260 con 0x7f83380134e0 2021-03-09T10:02:50.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.822+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6351 cs=0 l=1 rev1=1 rx=0x7f8338039ca0 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:50.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.826+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402b220 con 0x7f83380134e0 2021-03-09T10:02:50.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.826+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834406ce90 con 0x7f83380134e0 2021-03-09T10:02:50.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:50.826+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400f990 con 0x7f83380134e0 2021-03-09T10:02:51.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:02:50 smithi146 bash[13136]: debug 2021-03-09T10:02:50.821+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18185 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:02:55.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:02:55.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:55.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6351 cs=0 l=1 rev1=1 rx=0x7f8338039ca0 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:55.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6351 cs=0 l=1 rev1=1 rx=0x7f8338039ca0 tx=0x7f8344004520).stop 2021-03-09T10:02:55.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:55.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:55.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338011ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:55.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a990 con 0x7f8324003400 2021-03-09T10:02:55.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8338015d30 2021-03-09T10:02:55.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008420 con 0x7f8324005e00 2021-03-09T10:02:55.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:02:55.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:55.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:02:55.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:55.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.822+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338011ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:55.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.826+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:55.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.826+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:55.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.826+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:55.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.826+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:55.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.826+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b120 con 0x7f8338015d30 2021-03-09T10:02:55.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.826+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338011ee0 secure :-1 s=READY pgs=6358 cs=0 l=1 rev1=1 rx=0x7f8344011040 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:55.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.830+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344012920 con 0x7f8338015d30 2021-03-09T10:02:55.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.830+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834406ce90 con 0x7f8338015d30 2021-03-09T10:02:55.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:02:55.834+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405d1d0 con 0x7f8338015d30 2021-03-09T10:02:56.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:02:55 smithi146 bash[13136]: debug 2021-03-09T10:02:55.821+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18210 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:03:00.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338011ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:03:00.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338011ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:00.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338011ee0 secure :-1 s=READY pgs=6358 cs=0 l=1 rev1=1 rx=0x7f8344011040 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:00.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338011ee0 secure :-1 s=READY pgs=6358 cs=0 l=1 rev1=1 rx=0x7f8344011040 tx=0x7f8344004520).stop 2021-03-09T10:03:00.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338011ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:00.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338011ee0 unknown :-1 s=CLOSED pgs=6358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:00.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:00.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:00.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:00.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324003400 2021-03-09T10:03:00.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065d20 con 0x7f83380134e0 2021-03-09T10:03:00.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d790 con 0x7f8324005e00 2021-03-09T10:03:00.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:03:00.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:00.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:03:00.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:00.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.826+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:00.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.830+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:00.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.830+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:00.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.830+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:00.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.830+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:00.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.830+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801d8d0 con 0x7f83380134e0 2021-03-09T10:03:00.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.830+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6367 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c07c040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:00.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.830+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01a680 con 0x7f83380134e0 2021-03-09T10:03:00.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.830+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f83380134e0 2021-03-09T10:03:00.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:00.834+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c005240 con 0x7f83380134e0 2021-03-09T10:03:01.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:03:00 smithi146 bash[13136]: debug 2021-03-09T10:03:00.825+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18235 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:03:05.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:03:05.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:05.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6367 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c07c040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:05.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6367 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c07c040).stop 2021-03-09T10:03:05.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:05.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:05.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:05.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:05.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:05.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008420 con 0x7f8324003400 2021-03-09T10:03:05.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8338015d30 2021-03-09T10:03:05.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065d20 con 0x7f8324005e00 2021-03-09T10:03:05.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:03:05.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:05.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:03:05.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:05.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:05.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:05.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:05.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:05.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:05.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.830+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338060d10 con 0x7f8338015d30 2021-03-09T10:03:05.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.834+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d730 secure :-1 s=READY pgs=6375 cs=0 l=1 rev1=1 rx=0x7f8334064ac0 tx=0x7f83340041b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:05.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.834+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401f510 con 0x7f8338015d30 2021-03-09T10:03:05.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.834+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833400d130 con 0x7f8338015d30 2021-03-09T10:03:05.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:05.838+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340077b0 con 0x7f8338015d30 2021-03-09T10:03:06.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:03:05 smithi146 bash[13136]: debug 2021-03-09T10:03:05.828+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18261 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:03:10.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:03:10.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:10.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d730 secure :-1 s=READY pgs=6375 cs=0 l=1 rev1=1 rx=0x7f8334064ac0 tx=0x7f83340041b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:10.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d730 secure :-1 s=READY pgs=6375 cs=0 l=1 rev1=1 rx=0x7f8334064ac0 tx=0x7f83340041b0).stop 2021-03-09T10:03:10.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800d730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:10.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d730 unknown :-1 s=CLOSED pgs=6375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:10.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:10.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:10.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:10.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008420 con 0x7f8324003400 2021-03-09T10:03:10.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380263e0 con 0x7f83380134e0 2021-03-09T10:03:10.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021780 con 0x7f8324005e00 2021-03-09T10:03:10.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:03:10.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:03:10.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:10.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:10.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:10.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:10.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:10.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:10.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:10.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.833+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f83380134e0 2021-03-09T10:03:10.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.837+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6388 cs=0 l=1 rev1=1 rx=0x7f833c072ff0 tx=0x7f833c07c040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:10.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.837+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018e40 con 0x7f83380134e0 2021-03-09T10:03:10.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.841+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c057cf0 con 0x7f83380134e0 2021-03-09T10:03:10.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:10.841+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0050f0 con 0x7f83380134e0 2021-03-09T10:03:11.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:03:10 smithi146 bash[13136]: debug 2021-03-09T10:03:10.832+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18286 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:03:15.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:03:15.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:15.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6388 cs=0 l=1 rev1=1 rx=0x7f833c072ff0 tx=0x7f833c07c040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:15.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6388 cs=0 l=1 rev1=1 rx=0x7f833c072ff0 tx=0x7f833c07c040).stop 2021-03-09T10:03:15.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:15.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:15.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:15.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:15.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:15.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324003400 2021-03-09T10:03:15.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e3a0 con 0x7f8338015d30 2021-03-09T10:03:15.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:03:15.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d3d0 con 0x7f8324005e00 2021-03-09T10:03:15.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:15.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:03:15.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:15.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:15.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:15.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:15.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:15.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:15.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f8338015d30 2021-03-09T10:03:15.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.837+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d730 secure :-1 s=READY pgs=6393 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:15.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.841+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f8338015d30 2021-03-09T10:03:15.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.841+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338015d30 2021-03-09T10:03:15.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:15.841+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440791e0 con 0x7f8338015d30 2021-03-09T10:03:16.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:03:15 smithi146 bash[13136]: debug 2021-03-09T10:03:15.836+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18312 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:03:20.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.837+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:03:20.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.837+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:20.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.837+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d730 secure :-1 s=READY pgs=6393 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:20.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.837+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d730 secure :-1 s=READY pgs=6393 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344004520).stop 2021-03-09T10:03:20.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.837+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800d730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:20.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.837+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d730 unknown :-1 s=CLOSED pgs=6393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:20.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.837+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:20.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.837+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:20.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.837+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:20.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.837+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a690 con 0x7f8324003400 2021-03-09T10:03:20.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.837+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d3d0 con 0x7f83380134e0 2021-03-09T10:03:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.837+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007710 con 0x7f8324005e00 2021-03-09T10:03:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.837+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:03:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.837+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:03:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.837+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:20.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.837+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.841+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.841+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.841+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:20.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.841+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:20.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.841+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:20.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.841+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338025770 con 0x7f83380134e0 2021-03-09T10:03:20.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.841+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6401 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c057e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:20.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.841+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0042d0 con 0x7f83380134e0 2021-03-09T10:03:20.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.841+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0042d0 con 0x7f83380134e0 2021-03-09T10:03:20.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:20.845+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f83380134e0 2021-03-09T10:03:21.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:03:20 smithi146 bash[13136]: debug 2021-03-09T10:03:20.840+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18337 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:03:25.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:03:25.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:25.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6401 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c057e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:25.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6401 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c057e60).stop 2021-03-09T10:03:25.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:25.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:25.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:25.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:25.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:25.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002b20 con 0x7f8324003400 2021-03-09T10:03:25.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d3c0 con 0x7f8338015d30 2021-03-09T10:03:25.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380215f0 con 0x7f8324005e00 2021-03-09T10:03:25.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:03:25.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:25.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:25.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:25.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:25.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:25.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:03:25.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:25.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:25.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278f0 con 0x7f8338015d30 2021-03-09T10:03:25.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802cd20 secure :-1 s=READY pgs=6410 cs=0 l=1 rev1=1 rx=0x7f8344004690 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:25.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404ed80 con 0x7f8338015d30 2021-03-09T10:03:25.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.845+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f8338015d30 2021-03-09T10:03:25.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:25.849+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440791e0 con 0x7f8338015d30 2021-03-09T10:03:26.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:03:25 smithi146 bash[13136]: debug 2021-03-09T10:03:25.844+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18363 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:03:30.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:03:30.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:30.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802cd20 secure :-1 s=READY pgs=6410 cs=0 l=1 rev1=1 rx=0x7f8344004690 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:30.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802cd20 secure :-1 s=READY pgs=6410 cs=0 l=1 rev1=1 rx=0x7f8344004690 tx=0x7f8344004520).stop 2021-03-09T10:03:30.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802cd20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:30.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802cd20 unknown :-1 s=CLOSED pgs=6410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:30.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:30.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:30.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:30.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T10:03:30.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e3a0 con 0x7f83380134e0 2021-03-09T10:03:30.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324005e00 2021-03-09T10:03:30.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:03:30.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:30.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:03:30.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:30.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:30.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:30.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:30.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:30.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:30.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b120 con 0x7f83380134e0 2021-03-09T10:03:30.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6419 cs=0 l=1 rev1=1 rx=0x7f8338028840 tx=0x7f833c057e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:30.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.849+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c004210 con 0x7f83380134e0 2021-03-09T10:03:30.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.853+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c06ad80 con 0x7f83380134e0 2021-03-09T10:03:30.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:30.853+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01bb40 con 0x7f83380134e0 2021-03-09T10:03:31.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:03:30 smithi146 bash[13136]: debug 2021-03-09T10:03:30.844+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18388 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:03:35.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:03:35.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:35.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6419 cs=0 l=1 rev1=1 rx=0x7f8338028840 tx=0x7f833c057e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:35.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6419 cs=0 l=1 rev1=1 rx=0x7f8338028840 tx=0x7f833c057e60).stop 2021-03-09T10:03:35.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:35.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:35.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:35.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:35.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:35.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324003400 2021-03-09T10:03:35.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800cc80 con 0x7f8338015d30 2021-03-09T10:03:35.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324005e00 2021-03-09T10:03:35.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:03:35.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:35.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:03:35.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:35.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.849+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:35.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.853+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:35.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.853+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:35.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.853+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:35.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.853+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:35.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.853+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338021730 con 0x7f8338015d30 2021-03-09T10:03:35.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.853+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802cd20 secure :-1 s=READY pgs=6427 cs=0 l=1 rev1=1 rx=0x7f833c042c00 tx=0x7f833c057e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:35.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.853+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c042380 con 0x7f8338015d30 2021-03-09T10:03:35.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.853+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0424e0 con 0x7f8338015d30 2021-03-09T10:03:35.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:35.857+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c2a0 con 0x7f8338015d30 2021-03-09T10:03:36.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:03:35 smithi146 bash[13136]: debug 2021-03-09T10:03:35.848+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18414 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:03:40.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:03:40.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:40.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802cd20 secure :-1 s=READY pgs=6427 cs=0 l=1 rev1=1 rx=0x7f833c042c00 tx=0x7f833c057e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:40.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802cd20 secure :-1 s=READY pgs=6427 cs=0 l=1 rev1=1 rx=0x7f833c042c00 tx=0x7f833c057e60).stop 2021-03-09T10:03:40.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802cd20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:40.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802cd20 unknown :-1 s=CLOSED pgs=6427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:40.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:40.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:40.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:40.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324003400 2021-03-09T10:03:40.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a1b0 con 0x7f83380134e0 2021-03-09T10:03:40.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002b20 con 0x7f8324005e00 2021-03-09T10:03:40.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:03:40.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:03:40.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:40.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:40.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.853+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:40.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.861+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:40.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.861+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:40.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.861+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:40.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.861+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:40.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.861+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380068e0 con 0x7f83380134e0 2021-03-09T10:03:40.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.861+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6440 cs=0 l=1 rev1=1 rx=0x7f833c061f00 tx=0x7f833c009f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:40.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.865+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c026c60 con 0x7f83380134e0 2021-03-09T10:03:40.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.865+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01a270 con 0x7f83380134e0 2021-03-09T10:03:40.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:40.865+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c026c60 con 0x7f83380134e0 2021-03-09T10:03:41.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:03:40 smithi146 bash[13136]: debug 2021-03-09T10:03:40.852+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18439 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:03:45.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:03:45.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:45.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6440 cs=0 l=1 rev1=1 rx=0x7f833c061f00 tx=0x7f833c009f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:45.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6440 cs=0 l=1 rev1=1 rx=0x7f833c061f00 tx=0x7f833c009f50).stop 2021-03-09T10:03:45.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:45.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:45.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:45.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:03:45.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:45.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:45.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:45.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380637a0 con 0x7f8324003400 2021-03-09T10:03:45.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d3c0 con 0x7f8338015d30 2021-03-09T10:03:45.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bb80 con 0x7f8324005e00 2021-03-09T10:03:45.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:03:45.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:45.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:45.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:45.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:45.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:45.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:45.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e660 con 0x7f8338015d30 2021-03-09T10:03:45.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.857+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338021080 secure :-1 s=READY pgs=6445 cs=0 l=1 rev1=1 rx=0x7f833c029ad0 tx=0x7f833c009f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:45.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.861+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01c510 con 0x7f8338015d30 2021-03-09T10:03:45.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.861+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c510 con 0x7f8338015d30 2021-03-09T10:03:45.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:45.865+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054c60 con 0x7f8338015d30 2021-03-09T10:03:46.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:03:45 smithi146 bash[13136]: debug 2021-03-09T10:03:45.856+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18465 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:03:50.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:03:50.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:50.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338021080 secure :-1 s=READY pgs=6445 cs=0 l=1 rev1=1 rx=0x7f833c029ad0 tx=0x7f833c009f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:50.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338021080 secure :-1 s=READY pgs=6445 cs=0 l=1 rev1=1 rx=0x7f833c029ad0 tx=0x7f833c009f50).stop 2021-03-09T10:03:50.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:50.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338021080 unknown :-1 s=CLOSED pgs=6445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:50.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:50.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:50.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:50.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:50.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:03:50.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:50.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324003400 2021-03-09T10:03:50.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c1d0 con 0x7f83380134e0 2021-03-09T10:03:50.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dea0 con 0x7f8324005e00 2021-03-09T10:03:50.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:03:50.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:50.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:50.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:50.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:50.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:50.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.861+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380647b0 con 0x7f83380134e0 2021-03-09T10:03:50.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.865+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6453 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f83340041b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:50.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.865+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334033280 con 0x7f83380134e0 2021-03-09T10:03:50.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.865+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334052450 con 0x7f83380134e0 2021-03-09T10:03:50.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:50.869+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404ea00 con 0x7f83380134e0 2021-03-09T10:03:51.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:03:50 smithi146 bash[13136]: debug 2021-03-09T10:03:50.860+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18490 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:03:55.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:03:55.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:55.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6453 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f83340041b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:55.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6453 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f83340041b0).stop 2021-03-09T10:03:55.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:55.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:55.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:55.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:55.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:55.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015850 con 0x7f8324003400 2021-03-09T10:03:55.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a1b0 con 0x7f8338015d30 2021-03-09T10:03:55.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dea0 con 0x7f8324005e00 2021-03-09T10:03:55.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:03:55.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:03:55.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:55.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:55.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:55.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:55.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:55.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:55.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:55.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800bdb0 con 0x7f8338015d30 2021-03-09T10:03:55.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.864+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338021080 secure :-1 s=READY pgs=6462 cs=0 l=1 rev1=1 rx=0x7f83440567b0 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:55.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.868+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405be80 con 0x7f8338015d30 2021-03-09T10:03:55.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.868+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f8338015d30 2021-03-09T10:03:55.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:03:55.872+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440562d0 con 0x7f8338015d30 2021-03-09T10:03:56.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:03:55 smithi146 bash[13136]: debug 2021-03-09T10:03:55.864+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18516 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:04:00.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.864+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:04:00.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.864+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:00.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.864+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338021080 secure :-1 s=READY pgs=6462 cs=0 l=1 rev1=1 rx=0x7f83440567b0 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:00.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.864+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338021080 secure :-1 s=READY pgs=6462 cs=0 l=1 rev1=1 rx=0x7f83440567b0 tx=0x7f8344004520).stop 2021-03-09T10:04:00.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.864+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:00.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.864+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338021080 unknown :-1 s=CLOSED pgs=6462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:00.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.864+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:00.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.864+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:00.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.864+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:00.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.864+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028400 con 0x7f8324003400 2021-03-09T10:04:00.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.864+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068f90 con 0x7f83380134e0 2021-03-09T10:04:00.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.864+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024c10 con 0x7f8324005e00 2021-03-09T10:04:00.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.868+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:04:00.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.868+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:00.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.868+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:04:00.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.868+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:00.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.868+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:00.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.868+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:00.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.868+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:00.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.868+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:00.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.868+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:00.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.868+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026400 con 0x7f83380134e0 2021-03-09T10:04:00.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.872+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6473 cs=0 l=1 rev1=1 rx=0x7f833c01e040 tx=0x7f833c009f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:00.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.872+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018320 con 0x7f83380134e0 2021-03-09T10:04:00.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.872+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f83380134e0 2021-03-09T10:04:00.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:00.872+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f83380134e0 2021-03-09T10:04:01.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:04:00 smithi146 bash[13136]: debug 2021-03-09T10:04:00.872+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18541 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:04:05.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.868+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:04:05.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.868+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:05.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.868+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6473 cs=0 l=1 rev1=1 rx=0x7f833c01e040 tx=0x7f833c009f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:05.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.868+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6473 cs=0 l=1 rev1=1 rx=0x7f833c01e040 tx=0x7f833c009f50).stop 2021-03-09T10:04:05.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.868+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:05.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.868+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:05.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.868+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380645b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:05.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.868+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:05.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.872+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:05.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.872+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:04:05.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.872+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:05.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.872+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:05.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.872+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea20 con 0x7f8324003400 2021-03-09T10:04:05.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.872+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8338015d30 2021-03-09T10:04:05.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.872+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015850 con 0x7f8324005e00 2021-03-09T10:04:05.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.872+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:04:05.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.872+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:05.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.872+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:05.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.872+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:05.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.872+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:05.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.872+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:05.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.872+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e660 con 0x7f8338015d30 2021-03-09T10:04:05.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.872+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380645b0 secure :-1 s=READY pgs=6479 cs=0 l=1 rev1=1 rx=0x7f8334026b80 tx=0x7f83340525c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:05.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.876+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401adb0 con 0x7f8338015d30 2021-03-09T10:04:05.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.876+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334060de0 con 0x7f8338015d30 2021-03-09T10:04:05.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:05.876+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404e290 con 0x7f8338015d30 2021-03-09T10:04:06.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:04:05 smithi146 bash[13136]: debug 2021-03-09T10:04:05.876+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18567 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:04:10.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.872+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380645b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:04:10.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.872+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380645b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:10.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.872+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380645b0 secure :-1 s=READY pgs=6479 cs=0 l=1 rev1=1 rx=0x7f8334026b80 tx=0x7f83340525c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:10.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.872+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380645b0 secure :-1 s=READY pgs=6479 cs=0 l=1 rev1=1 rx=0x7f8334026b80 tx=0x7f83340525c0).stop 2021-03-09T10:04:10.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.872+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380645b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:10.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.872+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380645b0 unknown :-1 s=CLOSED pgs=6479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:10.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.872+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:10.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.872+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:10.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.872+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:10.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.872+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ce80 con 0x7f8324003400 2021-03-09T10:04:10.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.872+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea20 con 0x7f83380134e0 2021-03-09T10:04:10.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.872+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068f90 con 0x7f8324005e00 2021-03-09T10:04:10.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.872+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:04:10.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.872+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:10.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.872+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:04:10.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.872+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:10.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.876+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:10.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.876+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:10.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.876+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:10.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.876+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:10.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.876+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:10.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.876+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028980 con 0x7f83380134e0 2021-03-09T10:04:10.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.876+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6488 cs=0 l=1 rev1=1 rx=0x7f833403bcd0 tx=0x7f83340525c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:10.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.880+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833403b8e0 con 0x7f83380134e0 2021-03-09T10:04:10.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.880+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403b700 con 0x7f83380134e0 2021-03-09T10:04:10.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:10.884+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404e920 con 0x7f83380134e0 2021-03-09T10:04:11.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:04:10 smithi146 bash[13136]: debug 2021-03-09T10:04:10.880+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18592 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:04:15.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.876+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:04:15.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.876+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:15.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.876+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6488 cs=0 l=1 rev1=1 rx=0x7f833403bcd0 tx=0x7f83340525c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:15.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.876+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6488 cs=0 l=1 rev1=1 rx=0x7f833403bcd0 tx=0x7f83340525c0).stop 2021-03-09T10:04:15.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.876+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:15.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.876+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:15.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.880+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:15.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.880+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:15.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.880+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:04:15.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.880+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:15.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.880+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:04:15.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.880+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:15.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.880+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338017720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:15.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.880+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ce80 con 0x7f8324003400 2021-03-09T10:04:15.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.880+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e840 con 0x7f8338015d30 2021-03-09T10:04:15.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.880+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c1d0 con 0x7f8324005e00 2021-03-09T10:04:15.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.880+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338017720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:15.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.884+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:15.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.884+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:15.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.884+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:15.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.884+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:15.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.884+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800dfc0 con 0x7f8338015d30 2021-03-09T10:04:15.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.884+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338017720 secure :-1 s=READY pgs=6501 cs=0 l=1 rev1=1 rx=0x7f833c061b10 tx=0x7f833c009f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:15.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.884+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c007980 con 0x7f8338015d30 2021-03-09T10:04:15.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.884+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c061d20 con 0x7f8338015d30 2021-03-09T10:04:15.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:15.888+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f8338015d30 2021-03-09T10:04:16.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:04:15 smithi146 bash[13136]: debug 2021-03-09T10:04:15.883+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18617 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:04:20.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.880+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338017720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:04:20.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.880+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338017720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:20.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.880+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338017720 secure :-1 s=READY pgs=6501 cs=0 l=1 rev1=1 rx=0x7f833c061b10 tx=0x7f833c009f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:20.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.880+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338017720 secure :-1 s=READY pgs=6501 cs=0 l=1 rev1=1 rx=0x7f833c061b10 tx=0x7f833c009f50).stop 2021-03-09T10:04:20.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.880+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338017720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:20.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.880+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338017720 unknown :-1 s=CLOSED pgs=6501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:20.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.880+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:20.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.880+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:20.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.880+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:04:20.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.880+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:20.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.880+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:20.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.880+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015bf0 con 0x7f8324003400 2021-03-09T10:04:20.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.880+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064160 con 0x7f83380134e0 2021-03-09T10:04:20.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.880+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026600 con 0x7f8324005e00 2021-03-09T10:04:20.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.880+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:04:20.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.880+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:20.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.884+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:20.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.884+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:20.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.884+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:20.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.884+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:20.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.884+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:20.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.884+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e660 con 0x7f83380134e0 2021-03-09T10:04:20.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.884+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6505 cs=0 l=1 rev1=1 rx=0x7f8344067000 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:20.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.888+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f83380134e0 2021-03-09T10:04:20.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.888+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404ecd0 con 0x7f83380134e0 2021-03-09T10:04:20.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:20.888+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834403bcd0 con 0x7f83380134e0 2021-03-09T10:04:21.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:04:20 smithi146 bash[13136]: debug 2021-03-09T10:04:20.887+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18643 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:04:25.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:04:25.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:25.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6505 cs=0 l=1 rev1=1 rx=0x7f8344067000 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:25.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6505 cs=0 l=1 rev1=1 rx=0x7f8344067000 tx=0x7f8344004520).stop 2021-03-09T10:04:25.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:25.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:25.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:25.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:25.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:25.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T10:04:25.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e550 con 0x7f8338015d30 2021-03-09T10:04:25.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea20 con 0x7f8324005e00 2021-03-09T10:04:25.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:04:25.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:25.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:25.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:04:25.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:25.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:25.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:25.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:25.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:25.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e660 con 0x7f8338015d30 2021-03-09T10:04:25.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a4b0 secure :-1 s=READY pgs=6514 cs=0 l=1 rev1=1 rx=0x7f833c045580 tx=0x7f833c009f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:25.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c045820 con 0x7f8338015d30 2021-03-09T10:04:25.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.888+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c045820 con 0x7f8338015d30 2021-03-09T10:04:25.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:25.896+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338015d30 2021-03-09T10:04:26.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:04:25 smithi146 bash[13136]: debug 2021-03-09T10:04:25.887+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18668 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:04:30.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:04:30.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:30.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a4b0 secure :-1 s=READY pgs=6514 cs=0 l=1 rev1=1 rx=0x7f833c045580 tx=0x7f833c009f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:30.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a4b0 secure :-1 s=READY pgs=6514 cs=0 l=1 rev1=1 rx=0x7f833c045580 tx=0x7f833c009f50).stop 2021-03-09T10:04:30.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801a4b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:30.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a4b0 unknown :-1 s=CLOSED pgs=6514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:30.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:30.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:30.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:30.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015bf0 con 0x7f8324003400 2021-03-09T10:04:30.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802dcf0 con 0x7f83380134e0 2021-03-09T10:04:30.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e550 con 0x7f8324005e00 2021-03-09T10:04:30.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:04:30.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:30.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:04:30.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:30.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:30.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:30.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:30.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:30.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:30.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e840 con 0x7f83380134e0 2021-03-09T10:04:30.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6526 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:30.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.892+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344010ba0 con 0x7f83380134e0 2021-03-09T10:04:30.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.896+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f83380134e0 2021-03-09T10:04:30.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:30.896+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f83380134e0 2021-03-09T10:04:31.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:04:30 smithi146 bash[13136]: debug 2021-03-09T10:04:30.891+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18694 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:04:35.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:04:35.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:35.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6526 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:35.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6526 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344004520).stop 2021-03-09T10:04:35.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:35.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:35.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338007c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:35.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:35.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338007c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:35.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:35.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802b700 con 0x7f8324003400 2021-03-09T10:04:35.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8338015d30 2021-03-09T10:04:35.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024c10 con 0x7f8324005e00 2021-03-09T10:04:35.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:04:35.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:04:35.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:35.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:35.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:35.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.892+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:35.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.896+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:35.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.896+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:35.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.896+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e660 con 0x7f8338015d30 2021-03-09T10:04:35.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.896+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338007c10 secure :-1 s=READY pgs=6532 cs=0 l=1 rev1=1 rx=0x7f8334052590 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:35.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.896+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334016360 con 0x7f8338015d30 2021-03-09T10:04:35.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.896+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402db20 con 0x7f8338015d30 2021-03-09T10:04:35.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:35.900+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403b290 con 0x7f8338015d30 2021-03-09T10:04:36.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:04:35 smithi146 bash[13136]: debug 2021-03-09T10:04:35.895+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18719 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:04:40.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338007c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:04:40.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338007c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:40.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338007c10 secure :-1 s=READY pgs=6532 cs=0 l=1 rev1=1 rx=0x7f8334052590 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:40.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338007c10 secure :-1 s=READY pgs=6532 cs=0 l=1 rev1=1 rx=0x7f8334052590 tx=0x7f8334049000).stop 2021-03-09T10:04:40.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338007c10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:40.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338007c10 unknown :-1 s=CLOSED pgs=6532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:40.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:40.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:40.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:40.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380066e0 con 0x7f8324003400 2021-03-09T10:04:40.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fed0 con 0x7f83380134e0 2021-03-09T10:04:40.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e550 con 0x7f8324005e00 2021-03-09T10:04:40.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:04:40.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:40.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:40.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:04:40.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:40.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:40.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:40.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:40.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:40.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e660 con 0x7f83380134e0 2021-03-09T10:04:40.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.896+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6541 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334049000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:40.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.900+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833403b670 con 0x7f83380134e0 2021-03-09T10:04:40.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.900+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403b7d0 con 0x7f83380134e0 2021-03-09T10:04:40.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:40.904+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340077b0 con 0x7f83380134e0 2021-03-09T10:04:41.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:04:40 smithi146 bash[13136]: debug 2021-03-09T10:04:40.899+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18745 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:04:45.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:04:45.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:45.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6541 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:45.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6541 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334049000).stop 2021-03-09T10:04:45.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:45.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:45.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:45.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:45.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:45.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015bf0 con 0x7f8324003400 2021-03-09T10:04:45.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802dcf0 con 0x7f8338015d30 2021-03-09T10:04:45.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e550 con 0x7f8324005e00 2021-03-09T10:04:45.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:04:45.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:04:45.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:45.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:45.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:45.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:45.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:45.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:45.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:45.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e660 con 0x7f8338015d30 2021-03-09T10:04:45.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.899+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a3f0 secure :-1 s=READY pgs=6549 cs=0 l=1 rev1=1 rx=0x7f833403c050 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:45.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.903+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340261b0 con 0x7f8338015d30 2021-03-09T10:04:45.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.903+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340263c0 con 0x7f8338015d30 2021-03-09T10:04:45.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:45.903+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401adb0 con 0x7f8338015d30 2021-03-09T10:04:46.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:04:45 smithi146 bash[13136]: debug 2021-03-09T10:04:45.903+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18769 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:04:50.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.903+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:04:50.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.903+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:50.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.903+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a3f0 secure :-1 s=READY pgs=6549 cs=0 l=1 rev1=1 rx=0x7f833403c050 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:50.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.903+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a3f0 secure :-1 s=READY pgs=6549 cs=0 l=1 rev1=1 rx=0x7f833403c050 tx=0x7f83340482a0).stop 2021-03-09T10:04:50.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.903+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801a3f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:50.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.903+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a3f0 unknown :-1 s=CLOSED pgs=6549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:50.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.903+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:50.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.903+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:50.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.903+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:50.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.903+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fed0 con 0x7f8324003400 2021-03-09T10:04:50.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.903+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e550 con 0x7f83380134e0 2021-03-09T10:04:50.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.903+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068f90 con 0x7f8324005e00 2021-03-09T10:04:50.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.903+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:04:50.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.903+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:50.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.907+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:04:50.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.907+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:50.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.907+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:50.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.907+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:50.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.907+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:50.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.907+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:50.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.907+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:50.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.907+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e840 con 0x7f83380134e0 2021-03-09T10:04:50.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.907+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6558 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c009f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:50.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.911+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01c950 con 0x7f83380134e0 2021-03-09T10:04:50.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.911+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f83380134e0 2021-03-09T10:04:50.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:50.915+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f83380134e0 2021-03-09T10:04:51.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:04:50 smithi146 bash[13136]: debug 2021-03-09T10:04:50.907+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18796 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:04:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:04:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6558 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c009f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6558 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c009f50).stop 2021-03-09T10:04:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:55.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:55.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:55.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:55.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011d50 con 0x7f8324003400 2021-03-09T10:04:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802dcf0 con 0x7f8338015d30 2021-03-09T10:04:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fed0 con 0x7f8324005e00 2021-03-09T10:04:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:04:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:04:55.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:55.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:55.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.907+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:55.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.911+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:55.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.911+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:55.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.911+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:55.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.911+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:55.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.911+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380068a0 con 0x7f8338015d30 2021-03-09T10:04:55.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.911+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a3f0 secure :-1 s=READY pgs=6566 cs=0 l=1 rev1=1 rx=0x7f833c0544b0 tx=0x7f833c009f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:55.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.911+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338015d30 2021-03-09T10:04:55.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.911+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c004160 con 0x7f8338015d30 2021-03-09T10:04:55.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:04:55.919+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338015d30 2021-03-09T10:04:56.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:04:55 smithi146 bash[13136]: debug 2021-03-09T10:04:55.911+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18820 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:05:00.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.911+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:05:00.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.911+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:00.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.911+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a3f0 secure :-1 s=READY pgs=6566 cs=0 l=1 rev1=1 rx=0x7f833c0544b0 tx=0x7f833c009f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:00.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.911+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a3f0 secure :-1 s=READY pgs=6566 cs=0 l=1 rev1=1 rx=0x7f833c0544b0 tx=0x7f833c009f50).stop 2021-03-09T10:05:00.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.911+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801a3f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:00.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.911+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a3f0 unknown :-1 s=CLOSED pgs=6566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:00.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.911+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:00.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.911+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:00.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.911+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:00.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.911+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023a00 con 0x7f8324003400 2021-03-09T10:05:00.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.911+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806b170 con 0x7f83380134e0 2021-03-09T10:05:00.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.911+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d60 con 0x7f8324005e00 2021-03-09T10:05:00.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.911+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:05:00.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.911+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:00.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.911+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:00.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.915+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:05:00.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.915+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:00.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.915+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.915+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.915+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.915+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.915+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e660 con 0x7f83380134e0 2021-03-09T10:05:00.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.915+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6575 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:00.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.915+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340261b0 con 0x7f83380134e0 2021-03-09T10:05:00.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.915+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340263c0 con 0x7f83380134e0 2021-03-09T10:05:00.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:00.923+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403b290 con 0x7f83380134e0 2021-03-09T10:05:01.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:05:00 smithi146 bash[13136]: debug 2021-03-09T10:05:00.915+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18847 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:05:05.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:05:05.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:05.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6575 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:05.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6575 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f83340482a0).stop 2021-03-09T10:05:05.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:05.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:05.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:05.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:05.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:05.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fed0 con 0x7f8324003400 2021-03-09T10:05:05.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802b700 con 0x7f8338015d30 2021-03-09T10:05:05.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d60 con 0x7f8324005e00 2021-03-09T10:05:05.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:05:05.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:05.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:05:05.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:05.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.915+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:05.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.919+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:05.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.919+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:05.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.919+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:05.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.919+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:05.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.919+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338015ba0 con 0x7f8338015d30 2021-03-09T10:05:05.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.919+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801c430 secure :-1 s=READY pgs=6583 cs=0 l=1 rev1=1 rx=0x7f833c0077b0 tx=0x7f833c009f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:05.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.919+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04d240 con 0x7f8338015d30 2021-03-09T10:05:05.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.919+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c029d30 con 0x7f8338015d30 2021-03-09T10:05:05.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:05.923+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0252e0 con 0x7f8338015d30 2021-03-09T10:05:06.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:05:05 smithi146 bash[13136]: debug 2021-03-09T10:05:05.915+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18871 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:05:10.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.919+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:05:10.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.919+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:10.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.919+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801c430 secure :-1 s=READY pgs=6583 cs=0 l=1 rev1=1 rx=0x7f833c0077b0 tx=0x7f833c009f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:10.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.919+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801c430 secure :-1 s=READY pgs=6583 cs=0 l=1 rev1=1 rx=0x7f833c0077b0 tx=0x7f833c009f50).stop 2021-03-09T10:05:10.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.919+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:10.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.919+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801c430 unknown :-1 s=CLOSED pgs=6583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:10.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.919+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:10.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.919+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:10.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.919+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:05:10.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.919+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:10.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.919+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:05:10.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.919+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:10.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.919+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:10.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.919+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802dcf0 con 0x7f8324003400 2021-03-09T10:05:10.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.919+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d60 con 0x7f83380134e0 2021-03-09T10:05:10.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.919+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d600 con 0x7f8324005e00 2021-03-09T10:05:10.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.923+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:10.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.923+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:10.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.923+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:10.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.923+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:10.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.923+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:10.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.923+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e660 con 0x7f83380134e0 2021-03-09T10:05:10.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.923+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6595 cs=0 l=1 rev1=1 rx=0x7f833c064c60 tx=0x7f833c009f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:10.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.927+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0254f0 con 0x7f83380134e0 2021-03-09T10:05:10.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.927+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c02e350 con 0x7f83380134e0 2021-03-09T10:05:10.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:10.931+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f83380134e0 2021-03-09T10:05:11.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:05:10 smithi146 bash[13136]: debug 2021-03-09T10:05:10.919+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18898 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:05:15.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:05:15.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:15.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6595 cs=0 l=1 rev1=1 rx=0x7f833c064c60 tx=0x7f833c009f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:15.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6595 cs=0 l=1 rev1=1 rx=0x7f833c064c60 tx=0x7f833c009f50).stop 2021-03-09T10:05:15.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:15.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:15.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:15.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:15.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:05:15.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:15.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:15.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:05:15.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:15.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062340 con 0x7f8324003400 2021-03-09T10:05:15.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002b20 con 0x7f8338015d30 2021-03-09T10:05:15.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802b700 con 0x7f8324005e00 2021-03-09T10:05:15.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.923+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:15.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.927+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:15.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.927+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:15.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.927+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:15.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.927+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:15.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.927+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800de30 con 0x7f8338015d30 2021-03-09T10:05:15.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.927+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 secure :-1 s=READY pgs=6606 cs=0 l=1 rev1=1 rx=0x7f833c04d4a0 tx=0x7f833c009f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:15.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.927+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f8338015d30 2021-03-09T10:05:15.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.931+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f8338015d30 2021-03-09T10:05:15.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:15.931+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007820 con 0x7f8338015d30 2021-03-09T10:05:16.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:05:15 smithi146 bash[13136]: debug 2021-03-09T10:05:15.923+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18922 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:05:20.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338076600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:05:20.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338076600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:20.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 secure :-1 s=READY pgs=6606 cs=0 l=1 rev1=1 rx=0x7f833c04d4a0 tx=0x7f833c009f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:20.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 secure :-1 s=READY pgs=6606 cs=0 l=1 rev1=1 rx=0x7f833c04d4a0 tx=0x7f833c009f50).stop 2021-03-09T10:05:20.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338076600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:20.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 unknown :-1 s=CLOSED pgs=6606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:20.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:20.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:20.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:20.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fed0 con 0x7f8324003400 2021-03-09T10:05:20.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802dcf0 con 0x7f83380134e0 2021-03-09T10:05:20.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:05:20.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380169f0 con 0x7f8324005e00 2021-03-09T10:05:20.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:20.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:05:20.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:20.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:20.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:20.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:20.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:20.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.927+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:20.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.931+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800fb10 con 0x7f83380134e0 2021-03-09T10:05:20.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.931+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6610 cs=0 l=1 rev1=1 rx=0x7f8334060740 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:20.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.931+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401a210 con 0x7f83380134e0 2021-03-09T10:05:20.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.931+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401a370 con 0x7f83380134e0 2021-03-09T10:05:20.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:20.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833406d4d0 con 0x7f83380134e0 2021-03-09T10:05:21.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:05:20 smithi146 bash[13136]: debug 2021-03-09T10:05:20.927+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18949 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:05:25.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.927+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:05:25.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.927+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:25.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.927+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6610 cs=0 l=1 rev1=1 rx=0x7f8334060740 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:25.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.927+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6610 cs=0 l=1 rev1=1 rx=0x7f8334060740 tx=0x7f83340482a0).stop 2021-03-09T10:05:25.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.927+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:25.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.927+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:25.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.931+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338011ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:25.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.931+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:25.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.931+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:25.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.931+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324003400 2021-03-09T10:05:25.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.931+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cae0 con 0x7f8338015d30 2021-03-09T10:05:25.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.931+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d60 con 0x7f8324005e00 2021-03-09T10:05:25.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.931+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:05:25.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.931+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:05:25.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.931+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:25.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.931+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:25.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.931+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338011ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:25.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.931+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:25.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.931+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:25.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.931+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:25.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.931+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:25.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.931+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338015ba0 con 0x7f8338015d30 2021-03-09T10:05:25.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.935+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338011ff0 secure :-1 s=READY pgs=6620 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:25.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833403b370 con 0x7f8338015d30 2021-03-09T10:05:25.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403b370 con 0x7f8338015d30 2021-03-09T10:05:25.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:25.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833406d3f0 con 0x7f8338015d30 2021-03-09T10:05:26.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:05:25 smithi146 bash[13136]: debug 2021-03-09T10:05:25.931+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 18973 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:05:30.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338011ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:05:30.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338011ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:30.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338011ff0 secure :-1 s=READY pgs=6620 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:30.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338011ff0 secure :-1 s=READY pgs=6620 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f83340482a0).stop 2021-03-09T10:05:30.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338011ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:30.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338011ff0 unknown :-1 s=CLOSED pgs=6620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:30.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:30.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:30.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:30.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802dcf0 con 0x7f8324003400 2021-03-09T10:05:30.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e000 con 0x7f83380134e0 2021-03-09T10:05:30.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025860 con 0x7f8324005e00 2021-03-09T10:05:30.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:05:30.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:30.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:30.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:05:30.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:30.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:30.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:30.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:30.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:30.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e660 con 0x7f83380134e0 2021-03-09T10:05:30.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.934+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6627 cs=0 l=1 rev1=1 rx=0x7f8334003530 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:30.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.938+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334003370 con 0x7f83380134e0 2021-03-09T10:05:30.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.938+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334003140 con 0x7f83380134e0 2021-03-09T10:05:30.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:30.938+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334045110 con 0x7f83380134e0 2021-03-09T10:05:31.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:05:30 smithi146 bash[13136]: debug 2021-03-09T10:05:30.934+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19000 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:05:35.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.934+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:05:35.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.934+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:35.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.934+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6627 cs=0 l=1 rev1=1 rx=0x7f8334003530 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:35.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.934+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6627 cs=0 l=1 rev1=1 rx=0x7f8334003530 tx=0x7f83340482a0).stop 2021-03-09T10:05:35.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.934+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:35.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.934+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:35.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.934+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:35.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.934+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:35.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.934+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:35.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.934+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068f90 con 0x7f8324003400 2021-03-09T10:05:35.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.934+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f8338015d30 2021-03-09T10:05:35.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.934+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a380 con 0x7f8324005e00 2021-03-09T10:05:35.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.938+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:05:35.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.938+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:05:35.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.938+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:35.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.938+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:35.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.938+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:35.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.938+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:35.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.938+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:35.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.938+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:35.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.938+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:35.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.938+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338015ba0 con 0x7f8338015d30 2021-03-09T10:05:35.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.938+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d310 secure :-1 s=READY pgs=6635 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:35.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.942+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f8338015d30 2021-03-09T10:05:35.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.942+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338015d30 2021-03-09T10:05:35.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:35.946+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338015d30 2021-03-09T10:05:36.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:05:35 smithi146 bash[13136]: debug 2021-03-09T10:05:35.938+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19024 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:05:40.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:05:40.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:40.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d310 secure :-1 s=READY pgs=6635 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:40.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d310 secure :-1 s=READY pgs=6635 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004520).stop 2021-03-09T10:05:40.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800d310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:40.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d310 unknown :-1 s=CLOSED pgs=6635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:40.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:40.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:40.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:40.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e050 con 0x7f8324003400 2021-03-09T10:05:40.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d60 con 0x7f83380134e0 2021-03-09T10:05:40.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ee60 con 0x7f8324005e00 2021-03-09T10:05:40.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:05:40.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:40.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:05:40.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:40.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:40.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:40.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:40.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:40.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:40.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e660 con 0x7f83380134e0 2021-03-09T10:05:40.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.942+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6644 cs=0 l=1 rev1=1 rx=0x7f833404fcf0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:40.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.946+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404fd60 con 0x7f83380134e0 2021-03-09T10:05:40.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.946+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401a210 con 0x7f83380134e0 2021-03-09T10:05:40.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:40.950+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340261b0 con 0x7f83380134e0 2021-03-09T10:05:41.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:05:40 smithi146 bash[13136]: debug 2021-03-09T10:05:40.942+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19051 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:05:45.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:05:45.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:45.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6644 cs=0 l=1 rev1=1 rx=0x7f833404fcf0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:45.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6644 cs=0 l=1 rev1=1 rx=0x7f833404fcf0 tx=0x7f83340482a0).stop 2021-03-09T10:05:45.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:45.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:45.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:45.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:45.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:45.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:05:45.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:05:45.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:45.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:45.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:45.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d800 con 0x7f8324003400 2021-03-09T10:05:45.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028400 con 0x7f8338015d30 2021-03-09T10:05:45.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380267b0 con 0x7f8324005e00 2021-03-09T10:05:45.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:45.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:45.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:45.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:45.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e660 con 0x7f8338015d30 2021-03-09T10:05:45.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.946+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d310 secure :-1 s=READY pgs=6653 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c009f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:45.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.950+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c054af0 con 0x7f8338015d30 2021-03-09T10:05:45.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.950+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338015d30 2021-03-09T10:05:45.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:45.950+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c1a0 con 0x7f8338015d30 2021-03-09T10:05:46.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:05:45 smithi146 bash[13136]: debug 2021-03-09T10:05:45.946+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19075 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:05:50.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.946+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:05:50.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.946+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:50.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.946+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d310 secure :-1 s=READY pgs=6653 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c009f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:50.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.946+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d310 secure :-1 s=READY pgs=6653 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c009f50).stop 2021-03-09T10:05:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.946+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800d310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.946+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800d310 unknown :-1 s=CLOSED pgs=6653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.946+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.946+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.946+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:50.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.946+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d60 con 0x7f8324003400 2021-03-09T10:05:50.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.946+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002980 con 0x7f83380134e0 2021-03-09T10:05:50.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.946+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800de80 con 0x7f8324005e00 2021-03-09T10:05:50.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.946+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:05:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.946+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.946+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:05:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.946+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.950+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:50.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.950+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:50.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.950+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:50.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.950+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:50.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.950+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:50.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.950+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b120 con 0x7f83380134e0 2021-03-09T10:05:50.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.950+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6662 cs=0 l=1 rev1=1 rx=0x7f8338054950 tx=0x7f833c009f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:50.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.950+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f83380134e0 2021-03-09T10:05:50.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.950+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c004160 con 0x7f83380134e0 2021-03-09T10:05:50.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:50.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c024d10 con 0x7f83380134e0 2021-03-09T10:05:51.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:05:50 smithi146 bash[13136]: debug 2021-03-09T10:05:50.950+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19102 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:05:55.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.950+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:05:55.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.950+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:55.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.950+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6662 cs=0 l=1 rev1=1 rx=0x7f8338054950 tx=0x7f833c009f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:55.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.950+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6662 cs=0 l=1 rev1=1 rx=0x7f8338054950 tx=0x7f833c009f50).stop 2021-03-09T10:05:55.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.950+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:55.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.950+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:55.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.950+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:55.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.950+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380249b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:55.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.950+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:55.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.950+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324003400 2021-03-09T10:05:55.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.950+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380147f0 con 0x7f8338015d30 2021-03-09T10:05:55.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.950+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d180 con 0x7f8324005e00 2021-03-09T10:05:55.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.950+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:05:55.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.950+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:55.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.950+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:05:55.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.950+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:55.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.954+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:55.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.954+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:55.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.954+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:55.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.954+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:55.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.954+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:55.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.954+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a0b0 con 0x7f8338015d30 2021-03-09T10:05:55.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.954+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380249b0 secure :-1 s=READY pgs=6671 cs=0 l=1 rev1=1 rx=0x7f834402c040 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:55.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:05:55.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:05:55.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:05:55.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338015d30 2021-03-09T10:05:56.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:05:55 smithi146 bash[13136]: debug 2021-03-09T10:05:55.954+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19126 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:06:00.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.954+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:06:00.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.954+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:00.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.954+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380249b0 secure :-1 s=READY pgs=6671 cs=0 l=1 rev1=1 rx=0x7f834402c040 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:00.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.954+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380249b0 secure :-1 s=READY pgs=6671 cs=0 l=1 rev1=1 rx=0x7f834402c040 tx=0x7f8344004520).stop 2021-03-09T10:06:00.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380249b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:00.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.954+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380249b0 unknown :-1 s=CLOSED pgs=6671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:00.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.954+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:00.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.954+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:00.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.954+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:06:00.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.954+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:00.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.954+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:06:00.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.954+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:00.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.954+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:00.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380267b0 con 0x7f8324003400 2021-03-09T10:06:00.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d60 con 0x7f83380134e0 2021-03-09T10:06:00.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068fb0 con 0x7f8324005e00 2021-03-09T10:06:00.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.958+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:00.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.962+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:00.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:00.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.962+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:00.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:00.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.962+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338003470 con 0x7f83380134e0 2021-03-09T10:06:00.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6681 cs=0 l=1 rev1=1 rx=0x7f833c05c1e0 tx=0x7f833c009f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:00.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05c000 con 0x7f83380134e0 2021-03-09T10:06:00.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05c000 con 0x7f83380134e0 2021-03-09T10:06:00.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:00.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f83380134e0 2021-03-09T10:06:01.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:06:00 smithi146 bash[13136]: debug 2021-03-09T10:06:00.958+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19152 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:06:05.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:06:05.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:05.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6681 cs=0 l=1 rev1=1 rx=0x7f833c05c1e0 tx=0x7f833c009f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:05.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6681 cs=0 l=1 rev1=1 rx=0x7f833c05c1e0 tx=0x7f833c009f50).stop 2021-03-09T10:06:05.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:05.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:05.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:05.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380249b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:05.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:05.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024460 con 0x7f8324003400 2021-03-09T10:06:05.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380217a0 con 0x7f8338015d30 2021-03-09T10:06:05.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324005e00 2021-03-09T10:06:05.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:06:05.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:05.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:05.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:06:05.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.958+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:05.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.962+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:05.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.962+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:05.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.962+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:05.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.962+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:05.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.962+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e660 con 0x7f8338015d30 2021-03-09T10:06:05.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.962+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380249b0 secure :-1 s=READY pgs=6687 cs=0 l=1 rev1=1 rx=0x7f834404b940 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:05.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:06:05.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:06:05.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:05.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338015d30 2021-03-09T10:06:06.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:06:05 smithi146 bash[13136]: debug 2021-03-09T10:06:05.958+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19177 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:06:10.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:06:10.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:10.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380249b0 secure :-1 s=READY pgs=6687 cs=0 l=1 rev1=1 rx=0x7f834404b940 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:10.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380249b0 secure :-1 s=READY pgs=6687 cs=0 l=1 rev1=1 rx=0x7f834404b940 tx=0x7f8344004520).stop 2021-03-09T10:06:10.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380249b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:10.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380249b0 unknown :-1 s=CLOSED pgs=6687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:10.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:10.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:10.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:10.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d180 con 0x7f8324003400 2021-03-09T10:06:10.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f83380134e0 2021-03-09T10:06:10.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064ed0 con 0x7f8324005e00 2021-03-09T10:06:10.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:06:10.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:06:10.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:10.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:10.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:10.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:10.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:10.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:10.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:10.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b120 con 0x7f83380134e0 2021-03-09T10:06:10.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6697 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c009f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:10.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0296d0 con 0x7f83380134e0 2021-03-09T10:06:10.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c029900 con 0x7f83380134e0 2021-03-09T10:06:10.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:10.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f83380134e0 2021-03-09T10:06:11.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:06:10 smithi146 bash[13136]: debug 2021-03-09T10:06:10.962+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19203 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:06:15.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:06:15.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:15.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6697 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c009f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:15.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6697 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c009f50).stop 2021-03-09T10:06:15.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:15.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:15.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338025230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:15.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:15.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:15.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324003400 2021-03-09T10:06:15.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f8338015d30 2021-03-09T10:06:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a120 con 0x7f8324005e00 2021-03-09T10:06:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:06:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:06:15.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:15.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.966+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338025230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:15.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.970+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:15.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.970+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:15.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.970+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:15.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.970+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.970+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801cef0 con 0x7f8338015d30 2021-03-09T10:06:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.970+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338025230 secure :-1 s=READY pgs=6708 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:15.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404e5e0 con 0x7f8338015d30 2021-03-09T10:06:15.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.974+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404e7e0 con 0x7f8338015d30 2021-03-09T10:06:15.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:15.978+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404e360 con 0x7f8338015d30 2021-03-09T10:06:16.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:06:15 smithi146 bash[13136]: debug 2021-03-09T10:06:15.966+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19228 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:06:20.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338025230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:06:20.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338025230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:20.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338025230 secure :-1 s=READY pgs=6708 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:20.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338025230 secure :-1 s=READY pgs=6708 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).stop 2021-03-09T10:06:20.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338025230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:20.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338025230 unknown :-1 s=CLOSED pgs=6708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:20.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:20.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:20.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:20.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e240 con 0x7f8324003400 2021-03-09T10:06:20.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064ed0 con 0x7f83380134e0 2021-03-09T10:06:20.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324005e00 2021-03-09T10:06:20.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:06:20.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:20.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:06:20.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:20.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.969+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:20.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.973+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:20.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.973+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:20.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.973+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:20.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.973+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:20.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.973+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802d9b0 con 0x7f83380134e0 2021-03-09T10:06:20.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.973+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6717 cs=0 l=1 rev1=1 rx=0x7f83380652d0 tx=0x7f833c029a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:20.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c09a6d0 con 0x7f83380134e0 2021-03-09T10:06:20.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c09a6d0 con 0x7f83380134e0 2021-03-09T10:06:20.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:20.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c071240 con 0x7f83380134e0 2021-03-09T10:06:21.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:06:20 smithi146 bash[13136]: debug 2021-03-09T10:06:20.970+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19254 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:06:25.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:06:25.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:25.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6717 cs=0 l=1 rev1=1 rx=0x7f83380652d0 tx=0x7f833c029a70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:25.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6717 cs=0 l=1 rev1=1 rx=0x7f83380652d0 tx=0x7f833c029a70).stop 2021-03-09T10:06:25.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:25.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:25.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:25.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:25.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:25.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002980 con 0x7f8324003400 2021-03-09T10:06:25.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d180 con 0x7f8338015d30 2021-03-09T10:06:25.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a120 con 0x7f8324005e00 2021-03-09T10:06:25.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:06:25.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:25.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:06:25.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:25.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:25.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:25.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:25.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:25.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:25.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802d9b0 con 0x7f8338015d30 2021-03-09T10:06:25.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.973+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802dcf0 secure :-1 s=READY pgs=6722 cs=0 l=1 rev1=1 rx=0x7f8334052680 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:25.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833405c420 con 0x7f8338015d30 2021-03-09T10:06:25.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833405c580 con 0x7f8338015d30 2021-03-09T10:06:25.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:25.981+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833405c420 con 0x7f8338015d30 2021-03-09T10:06:26.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:06:25 smithi146 bash[13136]: debug 2021-03-09T10:06:25.974+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19279 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:06:30.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:06:30.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:30.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802dcf0 secure :-1 s=READY pgs=6722 cs=0 l=1 rev1=1 rx=0x7f8334052680 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:30.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802dcf0 secure :-1 s=READY pgs=6722 cs=0 l=1 rev1=1 rx=0x7f8334052680 tx=0x7f83340482a0).stop 2021-03-09T10:06:30.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802dcf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:30.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802dcf0 unknown :-1 s=CLOSED pgs=6722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:30.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:30.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:30.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:06:30.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:30.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:30.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e240 con 0x7f8324003400 2021-03-09T10:06:30.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007770 con 0x7f83380134e0 2021-03-09T10:06:30.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024460 con 0x7f8324005e00 2021-03-09T10:06:30.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:30.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:06:30.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:30.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:30.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:30.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:30.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:30.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380099d0 con 0x7f83380134e0 2021-03-09T10:06:30.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.977+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6731 cs=0 l=1 rev1=1 rx=0x7f8344056e10 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:30.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.981+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f83380134e0 2021-03-09T10:06:30.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.981+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f83380134e0 2021-03-09T10:06:30.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:30.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f83380134e0 2021-03-09T10:06:31.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:06:30 smithi146 bash[13136]: debug 2021-03-09T10:06:30.978+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19301 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:06:35.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.977+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:06:35.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.977+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:35.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.977+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6731 cs=0 l=1 rev1=1 rx=0x7f8344056e10 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:35.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.977+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6731 cs=0 l=1 rev1=1 rx=0x7f8344056e10 tx=0x7f8344004520).stop 2021-03-09T10:06:35.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:35.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.977+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:35.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.977+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:35.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.977+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:35.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.977+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338023c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:35.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324003400 2021-03-09T10:06:35.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002980 con 0x7f8338015d30 2021-03-09T10:06:35.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.977+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064ed0 con 0x7f8324005e00 2021-03-09T10:06:35.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.977+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:06:35.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.977+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:35.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338023c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:35.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.981+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:06:35.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.981+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:35.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.981+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:35.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:35.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.981+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:35.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:35.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.981+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f8338015d30 2021-03-09T10:06:35.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338023c00 secure :-1 s=READY pgs=6740 cs=0 l=1 rev1=1 rx=0x7f833c01af90 tx=0x7f833c09aa10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:35.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00f300 con 0x7f8338015d30 2021-03-09T10:06:35.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0426f0 con 0x7f8338015d30 2021-03-09T10:06:35.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:35.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c09aa80 con 0x7f8338015d30 2021-03-09T10:06:36.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:06:35 smithi146 bash[13136]: debug 2021-03-09T10:06:35.982+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19330 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:06:40.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.981+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338023c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:06:40.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.981+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338023c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:40.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338023c00 secure :-1 s=READY pgs=6740 cs=0 l=1 rev1=1 rx=0x7f833c01af90 tx=0x7f833c09aa10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:40.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338023c00 secure :-1 s=READY pgs=6740 cs=0 l=1 rev1=1 rx=0x7f833c01af90 tx=0x7f833c09aa10).stop 2021-03-09T10:06:40.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338023c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:40.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.985+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338023c00 unknown :-1 s=CLOSED pgs=6740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:40.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.985+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:40.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.985+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:40.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.985+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:40.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e240 con 0x7f8324003400 2021-03-09T10:06:40.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018000 con 0x7f83380134e0 2021-03-09T10:06:40.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324005e00 2021-03-09T10:06:40.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.985+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:06:40.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.985+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:06:40.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.985+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:40.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.985+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:40.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:40.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.985+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:40.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.985+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:40.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:40.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:40.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b7b0 con 0x7f83380134e0 2021-03-09T10:06:40.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6749 cs=0 l=1 rev1=1 rx=0x7f833800daa0 tx=0x7f833c09aa10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:40.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f83380134e0 2021-03-09T10:06:40.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f83380134e0 2021-03-09T10:06:40.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:40.993+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c026c60 con 0x7f83380134e0 2021-03-09T10:06:41.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:06:40 smithi146 bash[13136]: debug 2021-03-09T10:06:40.985+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19355 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:06:45.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:06:45.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:45.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6749 cs=0 l=1 rev1=1 rx=0x7f833800daa0 tx=0x7f833c09aa10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:45.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6749 cs=0 l=1 rev1=1 rx=0x7f833800daa0 tx=0x7f833c09aa10).stop 2021-03-09T10:06:45.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:45.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:45.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:45.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:45.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:45.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324003400 2021-03-09T10:06:45.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d3e0 con 0x7f8338015d30 2021-03-09T10:06:45.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002980 con 0x7f8324005e00 2021-03-09T10:06:45.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:06:45.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:45.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:45.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:06:46.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:46.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:46.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:46.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:46.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:46.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802bb80 con 0x7f8338015d30 2021-03-09T10:06:46.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 secure :-1 s=READY pgs=6757 cs=0 l=1 rev1=1 rx=0x7f833c007a40 tx=0x7f833c09aa10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:46.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c007ab0 con 0x7f8338015d30 2021-03-09T10:06:46.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c011c60 con 0x7f8338015d30 2021-03-09T10:06:46.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:45.993+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0070e0 con 0x7f8338015d30 2021-03-09T10:06:46.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:06:45 smithi146 bash[13136]: debug 2021-03-09T10:06:45.989+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19381 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:06:50.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338076600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:06:50.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338076600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:50.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 secure :-1 s=READY pgs=6757 cs=0 l=1 rev1=1 rx=0x7f833c007a40 tx=0x7f833c09aa10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:50.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 secure :-1 s=READY pgs=6757 cs=0 l=1 rev1=1 rx=0x7f833c007a40 tx=0x7f833c09aa10).stop 2021-03-09T10:06:50.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338076600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:50.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.989+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 unknown :-1 s=CLOSED pgs=6757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:50.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.989+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:50.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.989+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:50.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.989+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:50.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e240 con 0x7f8324003400 2021-03-09T10:06:50.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380245e0 con 0x7f83380134e0 2021-03-09T10:06:50.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.989+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324005e00 2021-03-09T10:06:50.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.989+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:06:50.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.989+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:50.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.989+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:06:50.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.989+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:50.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.993+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:50.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.993+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:50.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.993+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:50.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.993+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:51.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.993+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:51.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.993+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278f0 con 0x7f83380134e0 2021-03-09T10:06:51.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.993+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6767 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:51.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.993+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404eb80 con 0x7f83380134e0 2021-03-09T10:06:51.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340247f0 con 0x7f83380134e0 2021-03-09T10:06:51.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:50.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334007960 con 0x7f83380134e0 2021-03-09T10:06:51.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:06:50 smithi146 bash[13136]: debug 2021-03-09T10:06:50.993+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19406 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:06:55.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:06:55.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:55.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6767 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:55.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6767 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f83340482a0).stop 2021-03-09T10:06:55.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:55.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:56.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:56.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:56.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:56.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324003400 2021-03-09T10:06:56.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a7e0 con 0x7f8338015d30 2021-03-09T10:06:56.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007a60 con 0x7f8324005e00 2021-03-09T10:06:56.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:06:56.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:56.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:06:56.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:56.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:56.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:56.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:56.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:56.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:56.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068f60 con 0x7f8338015d30 2021-03-09T10:06:56.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:55.997+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 secure :-1 s=READY pgs=6777 cs=0 l=1 rev1=1 rx=0x7f8338039ca0 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:56.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:56.001+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:06:56.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:56.001+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:06:56.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:06:56.005+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834403bcd0 con 0x7f8338015d30 2021-03-09T10:06:56.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:06:56 smithi146 bash[13136]: debug 2021-03-09T10:06:55.997+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19433 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:07:01.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338076600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:07:01.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338076600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:01.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 secure :-1 s=READY pgs=6777 cs=0 l=1 rev1=1 rx=0x7f8338039ca0 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:01.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 secure :-1 s=READY pgs=6777 cs=0 l=1 rev1=1 rx=0x7f8338039ca0 tx=0x7f8344004520).stop 2021-03-09T10:07:01.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338076600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:01.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338076600 unknown :-1 s=CLOSED pgs=6777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:01.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:01.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:01.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:01.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a7e0 con 0x7f8324003400 2021-03-09T10:07:01.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c870 con 0x7f83380134e0 2021-03-09T10:07:01.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324005e00 2021-03-09T10:07:01.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:07:01.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:01.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:07:01.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:01.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:00.997+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:01.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:01.001+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:01.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:01.001+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:01.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:01.001+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338005e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:01.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:01.001+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338005e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:01.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:01.001+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801f970 con 0x7f83380134e0 2021-03-09T10:07:01.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:01.001+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6783 cs=0 l=1 rev1=1 rx=0x7f8338028e20 tx=0x7f833c09aa10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:01.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:01.001+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f83380134e0 2021-03-09T10:07:01.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:01.001+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c09aa80 con 0x7f83380134e0 2021-03-09T10:07:01.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:01.005+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f83380134e0 2021-03-09T10:07:01.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:07:01 smithi146 bash[13136]: debug 2021-03-09T10:07:01.001+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19458 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:07:06.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:07:06.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:06.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6783 cs=0 l=1 rev1=1 rx=0x7f8338028e20 tx=0x7f833c09aa10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:06.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6783 cs=0 l=1 rev1=1 rx=0x7f8338028e20 tx=0x7f833c09aa10).stop 2021-03-09T10:07:06.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:06.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:06.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380686d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:06.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:06.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:06.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a7e0 con 0x7f8324003400 2021-03-09T10:07:06.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8338015d30 2021-03-09T10:07:06.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f8324005e00 2021-03-09T10:07:06.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:07:06.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:07:06.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:06.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:06.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380686d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:06.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:06.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:06.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:06.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:06.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f8338015d30 2021-03-09T10:07:06.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380686d0 secure :-1 s=READY pgs=6792 cs=0 l=1 rev1=1 rx=0x7f833401b010 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:06.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404b9b0 con 0x7f8338015d30 2021-03-09T10:07:06.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833400ba50 con 0x7f8338015d30 2021-03-09T10:07:06.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:06.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404e220 con 0x7f8338015d30 2021-03-09T10:07:06.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:07:06 smithi146 bash[13136]: debug 2021-03-09T10:07:06.005+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19485 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:07:11.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380686d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:07:11.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380686d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:11.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380686d0 secure :-1 s=READY pgs=6792 cs=0 l=1 rev1=1 rx=0x7f833401b010 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:11.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380686d0 secure :-1 s=READY pgs=6792 cs=0 l=1 rev1=1 rx=0x7f833401b010 tx=0x7f83340482a0).stop 2021-03-09T10:07:11.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380686d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:11.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380686d0 unknown :-1 s=CLOSED pgs=6792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:11.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:11.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:11.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:11.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d510 con 0x7f8324003400 2021-03-09T10:07:11.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c870 con 0x7f83380134e0 2021-03-09T10:07:11.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067670 con 0x7f8324005e00 2021-03-09T10:07:11.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:07:11.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:11.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:07:11.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:11.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:11.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:11.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:11.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:11.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:11.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b110 con 0x7f83380134e0 2021-03-09T10:07:11.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.008+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6800 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:11.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f83380134e0 2021-03-09T10:07:11.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f83380134e0 2021-03-09T10:07:11.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:11.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344066b10 con 0x7f83380134e0 2021-03-09T10:07:11.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:07:11 smithi146 bash[13136]: debug 2021-03-09T10:07:11.005+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19510 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:07:16.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:07:16.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6800 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 secure :-1 s=READY pgs=6800 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004520).stop 2021-03-09T10:07:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380134e0 0x7f8338074100 unknown :-1 s=CLOSED pgs=6800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324003400 2021-03-09T10:07:16.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380093f0 con 0x7f8338015d30 2021-03-09T10:07:16.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380695c0 con 0x7f8324005e00 2021-03-09T10:07:16.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:07:16.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:16.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:16.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:07:16.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:16.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:16.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:16.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:16.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:16.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.008+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b120 con 0x7f8338015d30 2021-03-09T10:07:16.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.012+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 secure :-1 s=READY pgs=6809 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:16.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404a070 con 0x7f8338015d30 2021-03-09T10:07:16.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338015d30 2021-03-09T10:07:16.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:16.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334003220 con 0x7f8338015d30 2021-03-09T10:07:16.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:07:16 smithi146 bash[13136]: debug 2021-03-09T10:07:16.009+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19537 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:07:21.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:07:21.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:21.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 secure :-1 s=READY pgs=6809 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:21.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 secure :-1 s=READY pgs=6809 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f83340482a0).stop 2021-03-09T10:07:21.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380100b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:21.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 unknown :-1 s=CLOSED pgs=6809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:21.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:21.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:21.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:21.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011730 con 0x7f8324003400 2021-03-09T10:07:21.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8338027b20 2021-03-09T10:07:21.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380695c0 con 0x7f8324005e00 2021-03-09T10:07:21.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:07:21.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:21.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:07:21.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:21.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.012+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:21.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.016+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:21.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.016+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:21.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.016+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:21.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.016+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:21.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.016+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b120 con 0x7f8338027b20 2021-03-09T10:07:21.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.016+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 secure :-1 s=READY pgs=6818 cs=0 l=1 rev1=1 rx=0x7f8334016310 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:21.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340337c0 con 0x7f8338027b20 2021-03-09T10:07:21.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334023170 con 0x7f8338027b20 2021-03-09T10:07:21.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:21.020+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833400b980 con 0x7f8338027b20 2021-03-09T10:07:21.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:07:21 smithi146 bash[13136]: debug 2021-03-09T10:07:21.013+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19562 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:07:26.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:07:26.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:26.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 secure :-1 s=READY pgs=6818 cs=0 l=1 rev1=1 rx=0x7f8334016310 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:26.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 secure :-1 s=READY pgs=6818 cs=0 l=1 rev1=1 rx=0x7f8334016310 tx=0x7f83340482a0).stop 2021-03-09T10:07:26.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:26.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 unknown :-1 s=CLOSED pgs=6818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:26.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:26.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:26.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:26.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068f30 con 0x7f8324003400 2021-03-09T10:07:26.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011c10 con 0x7f8338015d30 2021-03-09T10:07:26.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cf60 con 0x7f8324005e00 2021-03-09T10:07:26.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:07:26.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:26.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:26.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:07:26.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:26.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:26.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:26.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:26.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:26.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.016+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b120 con 0x7f8338015d30 2021-03-09T10:07:26.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.020+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 secure :-1 s=READY pgs=6826 cs=0 l=1 rev1=1 rx=0x7f834402bc10 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:26.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.020+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:07:26.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.020+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:07:26.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:26.024+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834403bcd0 con 0x7f8338015d30 2021-03-09T10:07:26.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:07:26 smithi146 bash[13136]: debug 2021-03-09T10:07:26.017+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19589 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:07:31.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:07:31.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:31.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 secure :-1 s=READY pgs=6826 cs=0 l=1 rev1=1 rx=0x7f834402bc10 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:31.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 secure :-1 s=READY pgs=6826 cs=0 l=1 rev1=1 rx=0x7f834402bc10 tx=0x7f8344004520).stop 2021-03-09T10:07:31.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380100b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:31.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 unknown :-1 s=CLOSED pgs=6826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:31.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:31.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:31.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:31.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e300 con 0x7f8324003400 2021-03-09T10:07:31.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8338027b20 2021-03-09T10:07:31.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002b80 con 0x7f8324005e00 2021-03-09T10:07:31.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:07:31.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:31.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:07:31.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:31.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:31.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:31.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:31.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:31.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:31.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806efe0 con 0x7f8338027b20 2021-03-09T10:07:31.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 secure :-1 s=READY pgs=6834 cs=0 l=1 rev1=1 rx=0x7f834400f550 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:31.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338027b20 2021-03-09T10:07:31.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.024+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338027b20 2021-03-09T10:07:31.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:31.028+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405bd20 con 0x7f8338027b20 2021-03-09T10:07:31.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:07:31 smithi146 bash[13136]: debug 2021-03-09T10:07:31.021+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19614 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:07:36.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:07:36.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:36.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 secure :-1 s=READY pgs=6834 cs=0 l=1 rev1=1 rx=0x7f834400f550 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:36.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 secure :-1 s=READY pgs=6834 cs=0 l=1 rev1=1 rx=0x7f834400f550 tx=0x7f8344004520).stop 2021-03-09T10:07:36.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:36.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 unknown :-1 s=CLOSED pgs=6834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:36.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338014d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:36.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:36.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:36.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380068f0 con 0x7f8324003400 2021-03-09T10:07:36.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067670 con 0x7f8338015d30 2021-03-09T10:07:36.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002b80 con 0x7f8324005e00 2021-03-09T10:07:36.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:07:36.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:36.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:07:36.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:36.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338014d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:36.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:36.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:36.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:36.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:36.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801a630 con 0x7f8338015d30 2021-03-09T10:07:36.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338014d00 secure :-1 s=READY pgs=6843 cs=0 l=1 rev1=1 rx=0x7f833401b010 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:36.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8338015d30 2021-03-09T10:07:36.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.028+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8338015d30 2021-03-09T10:07:36.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:36.036+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338015d30 2021-03-09T10:07:36.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:07:36 smithi146 bash[13136]: debug 2021-03-09T10:07:36.025+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19641 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:07:41.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338014d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:07:41.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338014d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:41.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338014d00 secure :-1 s=READY pgs=6843 cs=0 l=1 rev1=1 rx=0x7f833401b010 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:41.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338014d00 secure :-1 s=READY pgs=6843 cs=0 l=1 rev1=1 rx=0x7f833401b010 tx=0x7f83340482a0).stop 2021-03-09T10:07:41.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338014d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:41.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338014d00 unknown :-1 s=CLOSED pgs=6843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:41.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:41.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:41.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:41.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324003400 2021-03-09T10:07:41.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e300 con 0x7f8338027b20 2021-03-09T10:07:41.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065080 con 0x7f8324005e00 2021-03-09T10:07:41.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:07:41.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:41.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:41.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:07:41.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:41.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:41.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:41.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:41.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:41.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.032+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806efe0 con 0x7f8338027b20 2021-03-09T10:07:41.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.036+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 secure :-1 s=READY pgs=6852 cs=0 l=1 rev1=1 rx=0x7f833802d480 tx=0x7f8344004520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:41.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.036+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338027b20 2021-03-09T10:07:41.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.036+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8338027b20 2021-03-09T10:07:41.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:41.040+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405bd20 con 0x7f8338027b20 2021-03-09T10:07:41.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:07:41 smithi146 bash[13136]: debug 2021-03-09T10:07:41.028+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19666 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:07:46.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:07:46.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:46.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 secure :-1 s=READY pgs=6852 cs=0 l=1 rev1=1 rx=0x7f833802d480 tx=0x7f8344004520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:46.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 secure :-1 s=READY pgs=6852 cs=0 l=1 rev1=1 rx=0x7f833802d480 tx=0x7f8344004520).stop 2021-03-09T10:07:46.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:46.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 unknown :-1 s=CLOSED pgs=6852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:46.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:46.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:46.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:46.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380068f0 con 0x7f8324003400 2021-03-09T10:07:46.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067670 con 0x7f8338015d30 2021-03-09T10:07:46.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380247d0 con 0x7f8324005e00 2021-03-09T10:07:46.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:07:46.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:46.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:07:46.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:46.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:46.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:46.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:46.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:46.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:46.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.032+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801a630 con 0x7f8338015d30 2021-03-09T10:07:46.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.036+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 secure :-1 s=READY pgs=6861 cs=0 l=1 rev1=1 rx=0x7f833c01b970 tx=0x7f833c09aa10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:46.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.036+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338015d30 2021-03-09T10:07:46.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.036+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f8338015d30 2021-03-09T10:07:46.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:46.040+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338015d30 2021-03-09T10:07:46.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:07:46 smithi146 bash[13136]: debug 2021-03-09T10:07:46.032+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19693 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:07:51.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:07:51.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:51.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 secure :-1 s=READY pgs=6861 cs=0 l=1 rev1=1 rx=0x7f833c01b970 tx=0x7f833c09aa10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:51.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 secure :-1 s=READY pgs=6861 cs=0 l=1 rev1=1 rx=0x7f833c01b970 tx=0x7f833c09aa10).stop 2021-03-09T10:07:51.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380100b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:51.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 unknown :-1 s=CLOSED pgs=6861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:51.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:51.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:51.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:51.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065080 con 0x7f8324003400 2021-03-09T10:07:51.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bbd0 con 0x7f8338027b20 2021-03-09T10:07:51.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380068f0 con 0x7f8324005e00 2021-03-09T10:07:51.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:07:51.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:51.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:51.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:07:51.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:51.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:51.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:51.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:51.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:51.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.036+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338069000 con 0x7f8338027b20 2021-03-09T10:07:51.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.040+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 secure :-1 s=READY pgs=6869 cs=0 l=1 rev1=1 rx=0x7f8344057900 tx=0x7f8344027000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:51.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.040+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338027b20 2021-03-09T10:07:51.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.040+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338027b20 2021-03-09T10:07:51.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:51.044+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344007740 con 0x7f8338027b20 2021-03-09T10:07:51.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:07:51 smithi146 bash[13136]: debug 2021-03-09T10:07:51.036+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19718 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:07:56.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:07:56.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:56.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 secure :-1 s=READY pgs=6869 cs=0 l=1 rev1=1 rx=0x7f8344057900 tx=0x7f8344027000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:56.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 secure :-1 s=READY pgs=6869 cs=0 l=1 rev1=1 rx=0x7f8344057900 tx=0x7f8344027000).stop 2021-03-09T10:07:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074100 unknown :-1 s=CLOSED pgs=6869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:56.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e300 con 0x7f8324003400 2021-03-09T10:07:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8338015d30 2021-03-09T10:07:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380068f0 con 0x7f8324005e00 2021-03-09T10:07:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:07:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:07:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.043+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:56.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.047+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:56.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.047+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.047+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.047+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.047+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338013180 con 0x7f8338015d30 2021-03-09T10:07:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.047+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 secure :-1 s=READY pgs=6879 cs=0 l=1 rev1=1 rx=0x7f834404bae0 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:56.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404b6f0 con 0x7f8338015d30 2021-03-09T10:07:56.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404b510 con 0x7f8338015d30 2021-03-09T10:07:56.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:07:56.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344036370 con 0x7f8338015d30 2021-03-09T10:07:56.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:07:56 smithi146 bash[13136]: debug 2021-03-09T10:07:56.040+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19744 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:08:01.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.043+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:08:01.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.043+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:01.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.043+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 secure :-1 s=READY pgs=6879 cs=0 l=1 rev1=1 rx=0x7f834404bae0 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:01.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.043+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 secure :-1 s=READY pgs=6879 cs=0 l=1 rev1=1 rx=0x7f834404bae0 tx=0x7f8344049b10).stop 2021-03-09T10:08:01.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380100b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:01.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380100b0 unknown :-1 s=CLOSED pgs=6879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:01.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:01.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:08:01.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:01.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380728f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:01.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f83380688d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:01.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324003400 2021-03-09T10:08:01.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067670 con 0x7f8338027b20 2021-03-09T10:08:01.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065080 con 0x7f8324005e00 2021-03-09T10:08:01.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f83380688d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:08:01.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f83380688d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:01.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:01.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:01.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:01.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f83380688d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:01.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f83380688d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:01.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380627c0 con 0x7f8338027b20 2021-03-09T10:08:01.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.047+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380728f0 secure :-1 s=READY pgs=6887 cs=0 l=1 rev1=1 rx=0x7f8344049d50 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:01.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400f390 con 0x7f8338027b20 2021-03-09T10:08:01.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400f1b0 con 0x7f8338027b20 2021-03-09T10:08:01.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:01.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344054190 con 0x7f8338027b20 2021-03-09T10:08:01.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:08:01 smithi146 bash[13136]: debug 2021-03-09T10:08:01.044+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19770 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:08:06.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.047+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f83380728f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:08:06.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.047+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f83380728f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:06.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.047+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380728f0 secure :-1 s=READY pgs=6887 cs=0 l=1 rev1=1 rx=0x7f8344049d50 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:06.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.047+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380728f0 secure :-1 s=READY pgs=6887 cs=0 l=1 rev1=1 rx=0x7f8344049d50 tx=0x7f8344049b10).stop 2021-03-09T10:08:06.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f83380728f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:06.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.047+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380728f0 unknown :-1 s=CLOSED pgs=6887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:06.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.047+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:06.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.047+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:06.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.047+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:06.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380068f0 con 0x7f8324003400 2021-03-09T10:08:06.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380239d0 con 0x7f8338015d30 2021-03-09T10:08:06.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324005e00 2021-03-09T10:08:06.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.051+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:08:06.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.051+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:08:06.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.051+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:06.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.051+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:06.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.051+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:06.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.051+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:06.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.051+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:06.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.051+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:06.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.051+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:06.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.051+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010f80 con 0x7f8338015d30 2021-03-09T10:08:06.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.051+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833806c690 secure :-1 s=READY pgs=6895 cs=0 l=1 rev1=1 rx=0x7f833c057450 tx=0x7f833c09aa10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:06.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338015d30 2021-03-09T10:08:06.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d2d0 con 0x7f8338015d30 2021-03-09T10:08:06.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:06.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055270 con 0x7f8338015d30 2021-03-09T10:08:06.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:08:06 smithi146 bash[13136]: debug 2021-03-09T10:08:06.048+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19796 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:08:11.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.051+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:08:11.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.051+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:11.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.051+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833806c690 secure :-1 s=READY pgs=6895 cs=0 l=1 rev1=1 rx=0x7f833c057450 tx=0x7f833c09aa10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:11.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.051+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833806c690 secure :-1 s=READY pgs=6895 cs=0 l=1 rev1=1 rx=0x7f833c057450 tx=0x7f833c09aa10).stop 2021-03-09T10:08:11.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833806c690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:11.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.051+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833806c690 unknown :-1 s=CLOSED pgs=6895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:11.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.051+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:11.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.051+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:11.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.051+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:11.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007090 con 0x7f8324003400 2021-03-09T10:08:11.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d510 con 0x7f8338027b20 2021-03-09T10:08:11.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065080 con 0x7f8324005e00 2021-03-09T10:08:11.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.051+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:08:11.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.051+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:11.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.051+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:08:11.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.051+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:11.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:11.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:11.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:11.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:11.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:11.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380127a0 con 0x7f8338027b20 2021-03-09T10:08:11.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076e00 secure :-1 s=READY pgs=6904 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c09aa10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:11.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c036520 con 0x7f8338027b20 2021-03-09T10:08:11.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c036340 con 0x7f8338027b20 2021-03-09T10:08:11.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:11.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f8338027b20 2021-03-09T10:08:11.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:08:11 smithi146 bash[13136]: debug 2021-03-09T10:08:11.052+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19822 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:08:16.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338076e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:08:16.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338076e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:16.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076e00 secure :-1 s=READY pgs=6904 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c09aa10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:16.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076e00 secure :-1 s=READY pgs=6904 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c09aa10).stop 2021-03-09T10:08:16.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338076e00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:16.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.055+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076e00 unknown :-1 s=CLOSED pgs=6904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:16.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.055+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:16.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.055+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:16.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.055+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:16.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e300 con 0x7f8324003400 2021-03-09T10:08:16.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380068f0 con 0x7f8338015d30 2021-03-09T10:08:16.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d110 con 0x7f8324005e00 2021-03-09T10:08:16.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.055+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:08:16.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.055+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:16.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.055+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:08:16.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.055+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:16.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.059+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:16.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.059+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:16.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.059+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:16.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.059+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:16.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.059+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:16.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.059+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806efe0 con 0x7f8338015d30 2021-03-09T10:08:16.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.059+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 secure :-1 s=READY pgs=6913 cs=0 l=1 rev1=1 rx=0x7f833c026a80 tx=0x7f833c09aa10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:16.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c062f10 con 0x7f8338015d30 2021-03-09T10:08:16.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0544f0 con 0x7f8338015d30 2021-03-09T10:08:16.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:16.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f8338015d30 2021-03-09T10:08:16.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:08:16 smithi146 bash[13136]: debug 2021-03-09T10:08:16.056+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19848 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:08:21.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.059+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338074900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:08:21.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.059+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338074900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:21.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.059+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 secure :-1 s=READY pgs=6913 cs=0 l=1 rev1=1 rx=0x7f833c026a80 tx=0x7f833c09aa10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:21.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.059+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 secure :-1 s=READY pgs=6913 cs=0 l=1 rev1=1 rx=0x7f833c026a80 tx=0x7f833c09aa10).stop 2021-03-09T10:08:21.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338074900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:21.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.059+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 unknown :-1 s=CLOSED pgs=6913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:21.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.059+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:21.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.059+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:21.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.059+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:21.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.059+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:08:21.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.059+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:21.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.059+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:21.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324003400 2021-03-09T10:08:21.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020d70 con 0x7f8338027b20 2021-03-09T10:08:21.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324005e00 2021-03-09T10:08:21.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.063+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:08:21.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.063+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:21.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.063+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:21.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.063+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:21.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.063+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:21.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.063+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:21.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.063+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380127a0 con 0x7f8338027b20 2021-03-09T10:08:21.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.063+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380100b0 secure :-1 s=READY pgs=6921 cs=0 l=1 rev1=1 rx=0x7f8334028410 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:21.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.067+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334063d40 con 0x7f8338027b20 2021-03-09T10:08:21.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.067+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402d6a0 con 0x7f8338027b20 2021-03-09T10:08:21.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:21.067+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833406d180 con 0x7f8338027b20 2021-03-09T10:08:21.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:08:21 smithi146 bash[13136]: debug 2021-03-09T10:08:21.060+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19874 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:08:26.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:08:26.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:26.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380100b0 secure :-1 s=READY pgs=6921 cs=0 l=1 rev1=1 rx=0x7f8334028410 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:26.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380100b0 secure :-1 s=READY pgs=6921 cs=0 l=1 rev1=1 rx=0x7f8334028410 tx=0x7f83340482a0).stop 2021-03-09T10:08:26.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f83380100b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:26.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380100b0 unknown :-1 s=CLOSED pgs=6921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:26.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:26.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:26.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:26.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008560 con 0x7f8324003400 2021-03-09T10:08:26.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d110 con 0x7f8338015d30 2021-03-09T10:08:26.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d510 con 0x7f8324005e00 2021-03-09T10:08:26.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:08:26.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:08:26.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:26.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:26.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.063+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:26.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.067+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:26.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.067+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:26.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.067+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:26.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.067+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:26.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.067+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338008000 con 0x7f8338015d30 2021-03-09T10:08:26.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.067+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 secure :-1 s=READY pgs=6930 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:26.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.071+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:08:26.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.071+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338015d30 2021-03-09T10:08:26.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:26.075+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338015d30 2021-03-09T10:08:26.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:08:26 smithi146 bash[13136]: debug 2021-03-09T10:08:26.064+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19899 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:08:31.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338074900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:08:31.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338074900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:31.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 secure :-1 s=READY pgs=6930 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:31.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 secure :-1 s=READY pgs=6930 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344049b10).stop 2021-03-09T10:08:31.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338074900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:31.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 unknown :-1 s=CLOSED pgs=6930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:31.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:31.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:31.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:31.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021970 con 0x7f8324003400 2021-03-09T10:08:31.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f9c0 con 0x7f8338027b20 2021-03-09T10:08:31.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026600 con 0x7f8324005e00 2021-03-09T10:08:31.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:08:31.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:31.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:08:31.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:31.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.067+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:31.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.071+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:31.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.071+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:31.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.071+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:31.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.071+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:31.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.071+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022220 con 0x7f8338027b20 2021-03-09T10:08:31.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.071+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380100b0 secure :-1 s=READY pgs=6941 cs=0 l=1 rev1=1 rx=0x7f833c007560 tx=0x7f833c054310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:31.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.075+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338027b20 2021-03-09T10:08:31.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.075+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338027b20 2021-03-09T10:08:31.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:31.079+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338027b20 2021-03-09T10:08:31.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:08:31 smithi146 bash[13136]: debug 2021-03-09T10:08:31.072+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19926 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:08:36.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:08:36.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:36.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380100b0 secure :-1 s=READY pgs=6941 cs=0 l=1 rev1=1 rx=0x7f833c007560 tx=0x7f833c054310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:36.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380100b0 secure :-1 s=READY pgs=6941 cs=0 l=1 rev1=1 rx=0x7f833c007560 tx=0x7f833c054310).stop 2021-03-09T10:08:36.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f83380100b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:36.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380100b0 unknown :-1 s=CLOSED pgs=6941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:36.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:36.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:36.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:36.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028ae0 con 0x7f8324003400 2021-03-09T10:08:36.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e300 con 0x7f8338015d30 2021-03-09T10:08:36.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021970 con 0x7f8324005e00 2021-03-09T10:08:36.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:08:36.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:36.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:08:36.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:36.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.071+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:36.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.075+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:36.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.075+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:36.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.075+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:36.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.075+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:36.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.075+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338067620 con 0x7f8338015d30 2021-03-09T10:08:36.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.075+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 secure :-1 s=READY pgs=6947 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:36.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.075+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:08:36.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.079+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:08:36.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:36.079+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338015d30 2021-03-09T10:08:36.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:08:36 smithi146 bash[13136]: debug 2021-03-09T10:08:36.072+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19951 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:08:41.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338074900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:08:41.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338074900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:41.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 secure :-1 s=READY pgs=6947 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:41.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 secure :-1 s=READY pgs=6947 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344049b10).stop 2021-03-09T10:08:41.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338074900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:41.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338074900 unknown :-1 s=CLOSED pgs=6947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:41.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:41.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833802db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:41.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:41.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011010 con 0x7f8324003400 2021-03-09T10:08:41.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b480 con 0x7f8338027b20 2021-03-09T10:08:41.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e300 con 0x7f8324005e00 2021-03-09T10:08:41.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:08:41.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:41.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:08:41.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:41.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.074+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833802db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:41.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.078+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:41.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.078+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:41.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.078+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:41.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.078+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:41.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.078+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338067620 con 0x7f8338027b20 2021-03-09T10:08:41.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.078+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833802db70 secure :-1 s=READY pgs=6957 cs=0 l=1 rev1=1 rx=0x7f834406a0b0 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:41.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404b370 con 0x7f8338027b20 2021-03-09T10:08:41.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344017640 con 0x7f8338027b20 2021-03-09T10:08:41.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:41.086+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ac90 con 0x7f8338027b20 2021-03-09T10:08:41.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:08:41 smithi146 bash[13136]: debug 2021-03-09T10:08:41.075+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 19978 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:08:46.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f833802db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:08:46.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f833802db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:46.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833802db70 secure :-1 s=READY pgs=6957 cs=0 l=1 rev1=1 rx=0x7f834406a0b0 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:46.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833802db70 secure :-1 s=READY pgs=6957 cs=0 l=1 rev1=1 rx=0x7f834406a0b0 tx=0x7f8344049b10).stop 2021-03-09T10:08:46.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f833802db70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:46.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833802db70 unknown :-1 s=CLOSED pgs=6957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:46.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:46.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:46.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:46.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012850 con 0x7f8324003400 2021-03-09T10:08:46.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016310 con 0x7f8338015d30 2021-03-09T10:08:46.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023e00 con 0x7f8324005e00 2021-03-09T10:08:46.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:08:46.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:46.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:08:46.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:46.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:46.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.078+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:46.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.082+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:46.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.082+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:46.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.082+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:46.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.082+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802cb40 con 0x7f8338015d30 2021-03-09T10:08:46.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.082+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748e0 secure :-1 s=READY pgs=6965 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:46.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833403ab20 con 0x7f8338015d30 2021-03-09T10:08:46.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402d6a0 con 0x7f8338015d30 2021-03-09T10:08:46.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:46.086+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402e630 con 0x7f8338015d30 2021-03-09T10:08:46.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:08:46 smithi146 bash[13136]: debug 2021-03-09T10:08:46.079+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20003 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:08:51.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:08:51.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:51.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748e0 secure :-1 s=READY pgs=6965 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:51.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748e0 secure :-1 s=READY pgs=6965 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f83340482a0).stop 2021-03-09T10:08:51.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380748e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:51.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748e0 unknown :-1 s=CLOSED pgs=6965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:51.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:51.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:51.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:51.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021970 con 0x7f8324003400 2021-03-09T10:08:51.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067670 con 0x7f8338027b20 2021-03-09T10:08:51.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024040 con 0x7f8324005e00 2021-03-09T10:08:51.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:08:51.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:51.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:51.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:08:51.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.082+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:51.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.086+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:51.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.086+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:51.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.086+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:51.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.086+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:51.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.086+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bc10 con 0x7f8338027b20 2021-03-09T10:08:51.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.086+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 secure :-1 s=READY pgs=6973 cs=0 l=1 rev1=1 rx=0x7f8334052680 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:51.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.086+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334068e10 con 0x7f8338027b20 2021-03-09T10:08:51.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.086+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334068e10 con 0x7f8338027b20 2021-03-09T10:08:51.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:51.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d610 con 0x7f8338027b20 2021-03-09T10:08:51.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:08:51 smithi146 bash[13136]: debug 2021-03-09T10:08:51.083+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20030 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:08:56.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:08:56.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:56.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 secure :-1 s=READY pgs=6973 cs=0 l=1 rev1=1 rx=0x7f8334052680 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:56.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 secure :-1 s=READY pgs=6973 cs=0 l=1 rev1=1 rx=0x7f8334052680 tx=0x7f83340482a0).stop 2021-03-09T10:08:56.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338076de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:56.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 unknown :-1 s=CLOSED pgs=6973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:56.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:56.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:56.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:56.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a990 con 0x7f8324003400 2021-03-09T10:08:56.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8338015d30 2021-03-09T10:08:56.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024040 con 0x7f8324005e00 2021-03-09T10:08:56.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:08:56.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:56.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:08:56.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:56.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:56.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:56.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:56.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:56.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:56.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.086+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380239b0 con 0x7f8338015d30 2021-03-09T10:08:56.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.090+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748e0 secure :-1 s=READY pgs=6982 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c054310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:56.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018410 con 0x7f8338015d30 2021-03-09T10:08:56.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338015d30 2021-03-09T10:08:56.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:08:56.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f8338015d30 2021-03-09T10:08:56.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:08:56 smithi146 bash[13136]: debug 2021-03-09T10:08:56.087+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20055 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:09:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:09:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748e0 secure :-1 s=READY pgs=6982 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c054310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748e0 secure :-1 s=READY pgs=6982 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c054310).stop 2021-03-09T10:09:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380748e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748e0 unknown :-1 s=CLOSED pgs=6982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833801ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005dc0 con 0x7f8324003400 2021-03-09T10:09:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800be60 con 0x7f8338027b20 2021-03-09T10:09:01.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067670 con 0x7f8324005e00 2021-03-09T10:09:01.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:09:01.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:01.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:09:01.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:01.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833801ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:01.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.090+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:01.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.094+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338021d10 con 0x7f8338027b20 2021-03-09T10:09:01.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.094+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833801ab80 secure :-1 s=READY pgs=6993 cs=0 l=1 rev1=1 rx=0x7f833c059040 tx=0x7f833c054310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:01.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f8338027b20 2021-03-09T10:09:01.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f8338027b20 2021-03-09T10:09:01.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:01.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c020a60 con 0x7f8338027b20 2021-03-09T10:09:01.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:09:01 smithi146 bash[13136]: debug 2021-03-09T10:09:01.091+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20082 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:09:06.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f833801ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:09:06.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f833801ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:06.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833801ab80 secure :-1 s=READY pgs=6993 cs=0 l=1 rev1=1 rx=0x7f833c059040 tx=0x7f833c054310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:06.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833801ab80 secure :-1 s=READY pgs=6993 cs=0 l=1 rev1=1 rx=0x7f833c059040 tx=0x7f833c054310).stop 2021-03-09T10:09:06.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f833801ab80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:06.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833801ab80 unknown :-1 s=CLOSED pgs=6993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:06.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:06.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:09:06.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:06.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:06.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:06.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d510 con 0x7f8324003400 2021-03-09T10:09:06.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024040 con 0x7f8338015d30 2021-03-09T10:09:06.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019a10 con 0x7f8324005e00 2021-03-09T10:09:06.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:09:06.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:06.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:06.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:06.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:06.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:06.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:06.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068480 con 0x7f8338015d30 2021-03-09T10:09:06.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.094+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 secure :-1 s=READY pgs=6999 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:06.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344072f10 con 0x7f8338015d30 2021-03-09T10:09:06.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f8338015d30 2021-03-09T10:09:06.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:06.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344010760 con 0x7f8338015d30 2021-03-09T10:09:06.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:09:06 smithi146 bash[13136]: debug 2021-03-09T10:09:06.095+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20107 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:09:11.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380748c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:09:11.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380748c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:11.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 secure :-1 s=READY pgs=6999 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:11.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 secure :-1 s=READY pgs=6999 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344049b10).stop 2021-03-09T10:09:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380748c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 unknown :-1 s=CLOSED pgs=6999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:11.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:09:11.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b480 con 0x7f8338027b20 2021-03-09T10:09:11.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005dc0 con 0x7f8324005e00 2021-03-09T10:09:11.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:09:11.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:11.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:09:11.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:11.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.098+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:11.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.102+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:11.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.102+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:11.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.102+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:11.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.102+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:11.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.102+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800cde0 con 0x7f8338027b20 2021-03-09T10:09:11.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.102+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 secure :-1 s=READY pgs=7009 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:11.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338027b20 2021-03-09T10:09:11.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000d20 con 0x7f8338027b20 2021-03-09T10:09:11.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:11.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334033cc0 con 0x7f8338027b20 2021-03-09T10:09:11.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:09:11 smithi146 bash[13136]: debug 2021-03-09T10:09:11.099+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20134 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:09:16.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:09:16.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:16.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 secure :-1 s=READY pgs=7009 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:16.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 secure :-1 s=READY pgs=7009 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).stop 2021-03-09T10:09:16.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338076de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:16.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 unknown :-1 s=CLOSED pgs=7009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:16.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:16.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:09:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d5d0 con 0x7f8338015d30 2021-03-09T10:09:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380086c0 con 0x7f8324005e00 2021-03-09T10:09:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:09:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:09:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.102+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.106+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.106+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:16.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.106+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:16.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.106+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:16.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.106+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b120 con 0x7f8338015d30 2021-03-09T10:09:16.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.106+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 secure :-1 s=READY pgs=7021 cs=0 l=1 rev1=1 rx=0x7f833c030040 tx=0x7f833c054310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:16.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f8338015d30 2021-03-09T10:09:16.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c036340 con 0x7f8338015d30 2021-03-09T10:09:16.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:16.110+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c061300 con 0x7f8338015d30 2021-03-09T10:09:16.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:09:16 smithi146 bash[13136]: debug 2021-03-09T10:09:16.103+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20159 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:09:21.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.102+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380748c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:09:21.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.102+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380748c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:21.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.102+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 secure :-1 s=READY pgs=7021 cs=0 l=1 rev1=1 rx=0x7f833c030040 tx=0x7f833c054310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:21.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.102+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 secure :-1 s=READY pgs=7021 cs=0 l=1 rev1=1 rx=0x7f833c030040 tx=0x7f833c054310).stop 2021-03-09T10:09:21.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380748c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:21.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 unknown :-1 s=CLOSED pgs=7021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:21.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833800c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:21.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:21.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:21.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:09:21.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019a10 con 0x7f8324003400 2021-03-09T10:09:21.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dce0 con 0x7f8338027b20 2021-03-09T10:09:21.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:09:21.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:21.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:09:21.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:21.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833800c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:21.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:21.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:21.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:21.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:21.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278f0 con 0x7f8338027b20 2021-03-09T10:09:21.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.106+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833800c560 secure :-1 s=READY pgs=7025 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:21.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.110+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833406d3b0 con 0x7f8338027b20 2021-03-09T10:09:21.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.110+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833406d5e0 con 0x7f8338027b20 2021-03-09T10:09:21.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:21.114+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404ec30 con 0x7f8338027b20 2021-03-09T10:09:21.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:09:21 smithi146 bash[13136]: debug 2021-03-09T10:09:21.107+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20186 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:09:26.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.106+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:09:26.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.106+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:26.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.106+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833800c560 secure :-1 s=READY pgs=7025 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:26.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.106+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833800c560 secure :-1 s=READY pgs=7025 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f83340482a0).stop 2021-03-09T10:09:26.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f833800c560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:26.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.106+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f833800c560 unknown :-1 s=CLOSED pgs=7025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:26.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.106+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:26.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.106+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:26.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.106+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:26.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019a10 con 0x7f8324003400 2021-03-09T10:09:26.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f6f0 con 0x7f8338015d30 2021-03-09T10:09:26.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324005e00 2021-03-09T10:09:26.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.110+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:09:26.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.110+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:26.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.110+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:26.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.110+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:09:26.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.110+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:26.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.110+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:26.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.110+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:26.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.110+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:26.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.110+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:26.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.110+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338063ff0 con 0x7f8338015d30 2021-03-09T10:09:26.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.110+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 secure :-1 s=READY pgs=7034 cs=0 l=1 rev1=1 rx=0x7f8334048000 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:26.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.114+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334057420 con 0x7f8338015d30 2021-03-09T10:09:26.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.114+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833400d590 con 0x7f8338015d30 2021-03-09T10:09:26.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:26.118+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d220 con 0x7f8338015d30 2021-03-09T10:09:26.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:09:26 smithi146 bash[13136]: debug 2021-03-09T10:09:26.111+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20211 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:09:31.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.109+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380748c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:09:31.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.109+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380748c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:31.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.109+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 secure :-1 s=READY pgs=7034 cs=0 l=1 rev1=1 rx=0x7f8334048000 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:31.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.109+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 secure :-1 s=READY pgs=7034 cs=0 l=1 rev1=1 rx=0x7f8334048000 tx=0x7f83340482a0).stop 2021-03-09T10:09:31.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.109+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380748c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:31.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.109+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380748c0 unknown :-1 s=CLOSED pgs=7034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:31.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.109+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:31.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.109+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:31.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.109+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:31.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.109+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010470 con 0x7f8324003400 2021-03-09T10:09:31.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.109+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024fa0 con 0x7f8338027b20 2021-03-09T10:09:31.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.109+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008870 con 0x7f8324005e00 2021-03-09T10:09:31.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.109+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:09:31.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.109+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:31.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.113+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:31.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.113+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:09:31.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.113+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:31.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.113+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:31.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.113+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:31.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.113+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:31.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.113+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:31.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.113+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380772d0 con 0x7f8338027b20 2021-03-09T10:09:31.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.113+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 secure :-1 s=READY pgs=7043 cs=0 l=1 rev1=1 rx=0x7f834404eb50 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:31.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.113+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f8338027b20 2021-03-09T10:09:31.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.113+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344010ad0 con 0x7f8338027b20 2021-03-09T10:09:31.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:31.117+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338027b20 2021-03-09T10:09:31.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:09:31 smithi146 bash[13136]: debug 2021-03-09T10:09:31.115+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20238 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:09:36.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:09:36.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:36.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 secure :-1 s=READY pgs=7043 cs=0 l=1 rev1=1 rx=0x7f834404eb50 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:36.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 secure :-1 s=READY pgs=7043 cs=0 l=1 rev1=1 rx=0x7f834404eb50 tx=0x7f8344049b10).stop 2021-03-09T10:09:36.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338076de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:36.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338076de0 unknown :-1 s=CLOSED pgs=7043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:36.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:36.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338072f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:36.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:36.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324003400 2021-03-09T10:09:36.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019a10 con 0x7f8338015d30 2021-03-09T10:09:36.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:09:36.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:09:36.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:36.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338072f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:36.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:09:36.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:36.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:36.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:36.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:36.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:36.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a530 con 0x7f8338015d30 2021-03-09T10:09:36.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.117+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338072f00 secure :-1 s=READY pgs=7051 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:36.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.121+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403b220 con 0x7f8338015d30 2021-03-09T10:09:36.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.121+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f8338015d30 2021-03-09T10:09:36.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:36.121+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440173d0 con 0x7f8338015d30 2021-03-09T10:09:36.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:09:36 smithi146 bash[13136]: debug 2021-03-09T10:09:36.119+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20263 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:09:41.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.121+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338072f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:09:41.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.121+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338072f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:41.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.121+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338072f00 secure :-1 s=READY pgs=7051 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:41.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.121+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338072f00 secure :-1 s=READY pgs=7051 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f8344049b10).stop 2021-03-09T10:09:41.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.121+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338072f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:41.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.121+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338072f00 unknown :-1 s=CLOSED pgs=7051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:41.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.121+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:41.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.121+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:41.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.121+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380153a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:41.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.121+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024fa0 con 0x7f8324003400 2021-03-09T10:09:41.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.121+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019a10 con 0x7f8338027b20 2021-03-09T10:09:41.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.121+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:09:41.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.121+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:09:41.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.121+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:41.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.121+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:09:41.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.121+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:41.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.125+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380153a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:41.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.125+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:41.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.125+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:41.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.125+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:41.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.125+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:41.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.125+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e290 con 0x7f8338027b20 2021-03-09T10:09:41.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.129+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380153a0 secure :-1 s=READY pgs=7062 cs=0 l=1 rev1=1 rx=0x7f833c079ff0 tx=0x7f833c054310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:41.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c004210 con 0x7f8338027b20 2021-03-09T10:09:41.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f8338027b20 2021-03-09T10:09:41.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:41.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c036b60 con 0x7f8338027b20 2021-03-09T10:09:41.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:09:41 smithi146 bash[13136]: debug 2021-03-09T10:09:41.123+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20289 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:09:46.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.121+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f83380153a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:09:46.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.121+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f83380153a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:46.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.121+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380153a0 secure :-1 s=READY pgs=7062 cs=0 l=1 rev1=1 rx=0x7f833c079ff0 tx=0x7f833c054310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:46.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.121+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380153a0 secure :-1 s=READY pgs=7062 cs=0 l=1 rev1=1 rx=0x7f833c079ff0 tx=0x7f833c054310).stop 2021-03-09T10:09:46.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.121+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f83380153a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:46.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.121+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380153a0 unknown :-1 s=CLOSED pgs=7062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:46.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.121+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:46.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.121+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:46.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.121+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:46.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.121+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:09:46.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.121+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025a10 con 0x7f8338015d30 2021-03-09T10:09:46.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.121+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019a10 con 0x7f8324005e00 2021-03-09T10:09:46.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.121+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:09:46.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.121+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:46.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.121+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:09:46.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.121+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:46.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.125+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:46.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.125+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:46.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.125+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:46.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.125+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:46.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.125+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:46.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.125+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a410 con 0x7f8338015d30 2021-03-09T10:09:46.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.125+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800dd60 secure :-1 s=READY pgs=7069 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c054310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:46.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f8338015d30 2021-03-09T10:09:46.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c007410 con 0x7f8338015d30 2021-03-09T10:09:46.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:46.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c020a60 con 0x7f8338015d30 2021-03-09T10:09:46.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:09:46 smithi146 bash[13136]: debug 2021-03-09T10:09:46.126+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20315 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:09:51.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:09:51.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:51.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800dd60 secure :-1 s=READY pgs=7069 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c054310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:51.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800dd60 secure :-1 s=READY pgs=7069 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c054310).stop 2021-03-09T10:09:51.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800dd60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:51.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800dd60 unknown :-1 s=CLOSED pgs=7069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:51.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:51.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:51.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:51.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024fa0 con 0x7f8324003400 2021-03-09T10:09:51.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019a10 con 0x7f8338027b20 2021-03-09T10:09:51.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324005e00 2021-03-09T10:09:51.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:09:51.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:51.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:09:51.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:51.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:51.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:51.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:51.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:51.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:51.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338063ff0 con 0x7f8338027b20 2021-03-09T10:09:51.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.129+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074f10 secure :-1 s=READY pgs=7078 cs=0 l=1 rev1=1 rx=0x7f833c0092c0 tx=0x7f833c007310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:51.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c029730 con 0x7f8338027b20 2021-03-09T10:09:51.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f8338027b20 2021-03-09T10:09:51.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:51.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338027b20 2021-03-09T10:09:51.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:09:51 smithi146 bash[13136]: debug 2021-03-09T10:09:51.130+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20341 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:09:56.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.129+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:09:56.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.129+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:56.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.129+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074f10 secure :-1 s=READY pgs=7078 cs=0 l=1 rev1=1 rx=0x7f833c0092c0 tx=0x7f833c007310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:56.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.129+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074f10 secure :-1 s=READY pgs=7078 cs=0 l=1 rev1=1 rx=0x7f833c0092c0 tx=0x7f833c007310).stop 2021-03-09T10:09:56.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:56.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.129+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074f10 unknown :-1 s=CLOSED pgs=7078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:56.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.129+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:56.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.129+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:56.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.129+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:56.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380267b0 con 0x7f8324003400 2021-03-09T10:09:56.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338015d30 2021-03-09T10:09:56.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024fa0 con 0x7f8324005e00 2021-03-09T10:09:56.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.129+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:09:56.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.129+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:56.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.133+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:09:56.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.133+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:56.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.133+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:56.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.133+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:56.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.133+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:56.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.133+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:56.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.133+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:56.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.133+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010fc0 con 0x7f8338015d30 2021-03-09T10:09:56.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.133+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069d10 secure :-1 s=READY pgs=7086 cs=0 l=1 rev1=1 rx=0x7f834403c000 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:56.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f8338015d30 2021-03-09T10:09:56.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402ded0 con 0x7f8338015d30 2021-03-09T10:09:56.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:09:56.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f8338015d30 2021-03-09T10:09:56.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:09:56 smithi146 bash[13136]: debug 2021-03-09T10:09:56.134+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20367 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:10:01.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:10:01.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:01.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069d10 secure :-1 s=READY pgs=7086 cs=0 l=1 rev1=1 rx=0x7f834403c000 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:01.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069d10 secure :-1 s=READY pgs=7086 cs=0 l=1 rev1=1 rx=0x7f834403c000 tx=0x7f8344049b10).stop 2021-03-09T10:10:01.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338069d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:01.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069d10 unknown :-1 s=CLOSED pgs=7086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:01.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:01.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:01.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:01.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801af60 con 0x7f8324003400 2021-03-09T10:10:01.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d6f0 con 0x7f8338027b20 2021-03-09T10:10:01.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010e80 con 0x7f8324005e00 2021-03-09T10:10:01.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:10:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:10:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.133+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.137+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.137+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380131a0 con 0x7f8338027b20 2021-03-09T10:10:01.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.137+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074f10 secure :-1 s=READY pgs=7095 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:01.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338027b20 2021-03-09T10:10:01.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334033cc0 con 0x7f8338027b20 2021-03-09T10:10:01.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:01.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334003140 con 0x7f8338027b20 2021-03-09T10:10:01.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:10:01 smithi146 bash[13136]: debug 2021-03-09T10:10:01.134+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20393 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:10:06.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:10:06.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:06.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074f10 secure :-1 s=READY pgs=7095 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:06.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074f10 secure :-1 s=READY pgs=7095 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f83340482a0).stop 2021-03-09T10:10:06.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:06.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074f10 unknown :-1 s=CLOSED pgs=7095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:06.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338066730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:06.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:06.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:06.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003080 con 0x7f8324003400 2021-03-09T10:10:06.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019a10 con 0x7f8338015d30 2021-03-09T10:10:06.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380267b0 con 0x7f8324005e00 2021-03-09T10:10:06.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:10:06.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:06.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:10:06.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:06.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338066730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:06.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:06.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:06.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:06.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:06.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.137+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800d6a0 con 0x7f8338015d30 2021-03-09T10:10:06.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.141+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338066730 secure :-1 s=READY pgs=7103 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:06.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334003350 con 0x7f8338015d30 2021-03-09T10:10:06.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340077b0 con 0x7f8338015d30 2021-03-09T10:10:06.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:06.145+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340314f0 con 0x7f8338015d30 2021-03-09T10:10:06.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:10:06 smithi146 bash[13136]: debug 2021-03-09T10:10:06.138+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20419 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:10:11.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338066730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:10:11.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338066730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:11.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338066730 secure :-1 s=READY pgs=7103 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:11.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338066730 secure :-1 s=READY pgs=7103 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).stop 2021-03-09T10:10:11.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338066730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:11.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338066730 unknown :-1 s=CLOSED pgs=7103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:11.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:11.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:11.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:11.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f330 con 0x7f8324003400 2021-03-09T10:10:11.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ce40 con 0x7f8338027b20 2021-03-09T10:10:11.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025a10 con 0x7f8324005e00 2021-03-09T10:10:11.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:10:11.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:10:11.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:11.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:11.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.141+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:11.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.149+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:11.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.149+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:11.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.149+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:11.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.149+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:11.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.149+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007dc0 con 0x7f8338027b20 2021-03-09T10:10:11.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.149+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 secure :-1 s=READY pgs=7115 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:11.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334027250 con 0x7f8338027b20 2021-03-09T10:10:11.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403a1f0 con 0x7f8338027b20 2021-03-09T10:10:11.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:11.153+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334046610 con 0x7f8338027b20 2021-03-09T10:10:11.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:10:11 smithi146 bash[13136]: debug 2021-03-09T10:10:11.142+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20444 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:10:16.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:10:16.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:16.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 secure :-1 s=READY pgs=7115 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:16.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 secure :-1 s=READY pgs=7115 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).stop 2021-03-09T10:10:16.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:16.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 unknown :-1 s=CLOSED pgs=7115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:16.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:16.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:16.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:16.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fed0 con 0x7f8324003400 2021-03-09T10:10:16.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338015d30 2021-03-09T10:10:16.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324005e00 2021-03-09T10:10:16.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:10:16.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:16.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:10:16.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:16.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.144+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:16.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.148+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:16.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.148+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:16.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.148+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:16.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.148+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:16.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.148+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338063ff0 con 0x7f8338015d30 2021-03-09T10:10:16.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.148+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338025e80 secure :-1 s=READY pgs=7125 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:16.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.152+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:10:16.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.152+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344078190 con 0x7f8338015d30 2021-03-09T10:10:16.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:16.152+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f8338015d30 2021-03-09T10:10:16.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:10:16 smithi146 bash[13136]: debug 2021-03-09T10:10:16.146+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20471 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:10:21.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:10:21.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:21.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338025e80 secure :-1 s=READY pgs=7125 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:21.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338025e80 secure :-1 s=READY pgs=7125 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344049b10).stop 2021-03-09T10:10:21.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:21.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338025e80 unknown :-1 s=CLOSED pgs=7125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:21.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:21.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:21.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:21.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:10:21.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:21.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023a00 con 0x7f8324003400 2021-03-09T10:10:21.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338027b20 2021-03-09T10:10:21.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324005e00 2021-03-09T10:10:21.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:21.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:10:21.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:21.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:21.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:21.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:21.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:21.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.148+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065440 con 0x7f8338027b20 2021-03-09T10:10:21.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.152+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 secure :-1 s=READY pgs=7130 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:21.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.152+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334057750 con 0x7f8338027b20 2021-03-09T10:10:21.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.152+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334031130 con 0x7f8338027b20 2021-03-09T10:10:21.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:21.156+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833400b980 con 0x7f8338027b20 2021-03-09T10:10:21.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:10:21 smithi146 bash[13136]: debug 2021-03-09T10:10:21.150+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20496 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:10:26.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:10:26.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:26.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 secure :-1 s=READY pgs=7130 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:26.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 secure :-1 s=READY pgs=7130 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f83340482a0).stop 2021-03-09T10:10:26.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:26.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 unknown :-1 s=CLOSED pgs=7130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:26.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:26.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:26.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:26.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fed0 con 0x7f8324003400 2021-03-09T10:10:26.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c230 con 0x7f8338015d30 2021-03-09T10:10:26.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:10:26.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:10:26.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:26.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:10:26.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:26.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:26.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:26.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:26.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:26.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:26.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068f80 con 0x7f8338015d30 2021-03-09T10:10:26.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.152+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802c100 secure :-1 s=READY pgs=7138 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:26.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.164+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334071ea0 con 0x7f8338015d30 2021-03-09T10:10:26.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.164+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334071ea0 con 0x7f8338015d30 2021-03-09T10:10:26.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:26.164+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334045d50 con 0x7f8338015d30 2021-03-09T10:10:26.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:10:26 smithi146 bash[13136]: debug 2021-03-09T10:10:26.154+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20523 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:10:31.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:10:31.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:31.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802c100 secure :-1 s=READY pgs=7138 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:31.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802c100 secure :-1 s=READY pgs=7138 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f83340482a0).stop 2021-03-09T10:10:31.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802c100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:31.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802c100 unknown :-1 s=CLOSED pgs=7138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:31.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:31.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:31.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:31.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:10:31.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:31.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f140 con 0x7f8324003400 2021-03-09T10:10:31.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338027b20 2021-03-09T10:10:31.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010690 con 0x7f8324005e00 2021-03-09T10:10:31.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:10:31.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:31.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:31.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:31.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:31.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:31.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:31.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.156+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800d6a0 con 0x7f8338027b20 2021-03-09T10:10:31.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.160+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 secure :-1 s=READY pgs=7149 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:31.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.160+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403cef0 con 0x7f8338027b20 2021-03-09T10:10:31.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.160+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f8338027b20 2021-03-09T10:10:31.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:31.164+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834403ca20 con 0x7f8338027b20 2021-03-09T10:10:31.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:10:31 smithi146 bash[13136]: debug 2021-03-09T10:10:31.158+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20548 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:10:36.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:10:36.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:36.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 secure :-1 s=READY pgs=7149 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:36.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 secure :-1 s=READY pgs=7149 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344049b10).stop 2021-03-09T10:10:36.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:36.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ef0 unknown :-1 s=CLOSED pgs=7149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:36.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:36.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:36.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380773f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:36.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:10:36.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:36.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:10:36.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:36.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b450 con 0x7f8324003400 2021-03-09T10:10:36.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ce40 con 0x7f8338015d30 2021-03-09T10:10:36.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324005e00 2021-03-09T10:10:36.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380773f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:36.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:36.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:36.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:36.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:36.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.160+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380239b0 con 0x7f8338015d30 2021-03-09T10:10:36.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.164+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380773f0 secure :-1 s=READY pgs=7156 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c007310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:36.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.164+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f8338015d30 2021-03-09T10:10:36.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.164+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f8338015d30 2021-03-09T10:10:36.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:36.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f8338015d30 2021-03-09T10:10:36.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:10:36 smithi146 bash[13136]: debug 2021-03-09T10:10:36.162+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20575 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:10:41.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380773f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:10:41.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380773f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:41.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380773f0 secure :-1 s=READY pgs=7156 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c007310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:41.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380773f0 secure :-1 s=READY pgs=7156 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c007310).stop 2021-03-09T10:10:41.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380773f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:41.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380773f0 unknown :-1 s=CLOSED pgs=7156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:41.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:41.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:41.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:41.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324003400 2021-03-09T10:10:41.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012880 con 0x7f8338027b20 2021-03-09T10:10:41.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010690 con 0x7f8324005e00 2021-03-09T10:10:41.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:10:41.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:41.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:10:41.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:41.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:41.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:41.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:41.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:41.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:41.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.164+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338019a10 con 0x7f8338027b20 2021-03-09T10:10:41.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.168+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 secure :-1 s=READY pgs=7166 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:41.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f8338027b20 2021-03-09T10:10:41.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402b220 con 0x7f8338027b20 2021-03-09T10:10:41.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:41.172+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338027b20 2021-03-09T10:10:41.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:10:41 smithi146 bash[13136]: debug 2021-03-09T10:10:41.166+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20600 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:10:46.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.164+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:10:46.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.164+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:46.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.164+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 secure :-1 s=READY pgs=7166 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:46.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.164+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 secure :-1 s=READY pgs=7166 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344049b10).stop 2021-03-09T10:10:46.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.164+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:46.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.164+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 unknown :-1 s=CLOSED pgs=7166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:46.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.164+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338061ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:46.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.164+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:46.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.168+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:10:46.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.168+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:46.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.164+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:46.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069700 con 0x7f8324003400 2021-03-09T10:10:46.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012740 con 0x7f8338015d30 2021-03-09T10:10:46.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ce40 con 0x7f8324005e00 2021-03-09T10:10:46.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.168+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338061ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:46.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.168+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:10:46.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.168+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:46.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.168+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:46.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.168+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:46.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.168+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:46.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.168+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:46.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.168+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005f70 con 0x7f8338015d30 2021-03-09T10:10:46.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.168+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338061ec0 secure :-1 s=READY pgs=7174 cs=0 l=1 rev1=1 rx=0x7f8334048000 tx=0x7f8334015fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:46.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.172+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334045ce0 con 0x7f8338015d30 2021-03-09T10:10:46.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.172+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334045e40 con 0x7f8338015d30 2021-03-09T10:10:46.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:46.172+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402e650 con 0x7f8338015d30 2021-03-09T10:10:46.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:10:46 smithi146 bash[13136]: debug 2021-03-09T10:10:46.170+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20627 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:10:51.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.168+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338061ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:10:51.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.168+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338061ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:51.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.168+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338061ec0 secure :-1 s=READY pgs=7174 cs=0 l=1 rev1=1 rx=0x7f8334048000 tx=0x7f8334015fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:51.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.168+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338061ec0 secure :-1 s=READY pgs=7174 cs=0 l=1 rev1=1 rx=0x7f8334048000 tx=0x7f8334015fa0).stop 2021-03-09T10:10:51.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338061ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:51.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.168+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338061ec0 unknown :-1 s=CLOSED pgs=7174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:51.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.168+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:51.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.168+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:51.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.168+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:51.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b450 con 0x7f8324003400 2021-03-09T10:10:51.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ce40 con 0x7f8338027b20 2021-03-09T10:10:51.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d460 con 0x7f8324005e00 2021-03-09T10:10:51.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.168+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:10:51.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.168+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:51.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.168+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:10:51.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.172+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:51.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.168+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:51.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.172+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:51.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.172+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:51.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.172+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:51.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.172+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:51.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.172+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800d6a0 con 0x7f8338027b20 2021-03-09T10:10:51.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.172+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 secure :-1 s=READY pgs=7181 cs=0 l=1 rev1=1 rx=0x7f8338066690 tx=0x7f833c007310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:51.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.172+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c010860 con 0x7f8338027b20 2021-03-09T10:10:51.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.172+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c010860 con 0x7f8338027b20 2021-03-09T10:10:51.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:51.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0252e0 con 0x7f8338027b20 2021-03-09T10:10:51.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:10:51 smithi146 bash[13136]: debug 2021-03-09T10:10:51.173+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20652 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:10:56.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:10:56.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:56.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 secure :-1 s=READY pgs=7181 cs=0 l=1 rev1=1 rx=0x7f8338066690 tx=0x7f833c007310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:56.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 secure :-1 s=READY pgs=7181 cs=0 l=1 rev1=1 rx=0x7f8338066690 tx=0x7f833c007310).stop 2021-03-09T10:10:56.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:56.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 unknown :-1 s=CLOSED pgs=7181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:56.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:56.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380243c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:56.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:56.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062890 con 0x7f8324003400 2021-03-09T10:10:56.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fed0 con 0x7f8338015d30 2021-03-09T10:10:56.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d460 con 0x7f8324005e00 2021-03-09T10:10:56.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:10:56.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:10:56.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:56.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:56.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380243c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:56.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:56.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:56.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:56.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:56.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ed10 con 0x7f8338015d30 2021-03-09T10:10:56.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380243c0 secure :-1 s=READY pgs=7190 cs=0 l=1 rev1=1 rx=0x7f834405bff0 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:56.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402cb40 con 0x7f8338015d30 2021-03-09T10:10:56.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402ded0 con 0x7f8338015d30 2021-03-09T10:10:56.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:10:56.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402cd80 con 0x7f8338015d30 2021-03-09T10:10:56.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:10:56 smithi146 bash[13136]: debug 2021-03-09T10:10:56.177+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20676 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:11:01.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380243c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:11:01.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380243c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:01.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380243c0 secure :-1 s=READY pgs=7190 cs=0 l=1 rev1=1 rx=0x7f834405bff0 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:01.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380243c0 secure :-1 s=READY pgs=7190 cs=0 l=1 rev1=1 rx=0x7f834405bff0 tx=0x7f8344049b10).stop 2021-03-09T10:11:01.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380243c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:01.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380243c0 unknown :-1 s=CLOSED pgs=7190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:01.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:01.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:01.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:01.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069700 con 0x7f8324003400 2021-03-09T10:11:01.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338027b20 2021-03-09T10:11:01.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012010 con 0x7f8324005e00 2021-03-09T10:11:01.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.176+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:01.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.180+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:11:01.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.180+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:01.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.180+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:11:01.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.180+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:01.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.180+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:01.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.180+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:01.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.180+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:01.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.180+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:01.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.180+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380192d0 con 0x7f8338027b20 2021-03-09T10:11:01.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.180+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 secure :-1 s=READY pgs=7199 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334015fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:01.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340079c0 con 0x7f8338027b20 2021-03-09T10:11:01.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833400b980 con 0x7f8338027b20 2021-03-09T10:11:01.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:01.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d220 con 0x7f8338027b20 2021-03-09T10:11:01.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:11:01 smithi146 bash[13136]: debug 2021-03-09T10:11:01.181+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20704 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:11:06.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:11:06.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:06.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 secure :-1 s=READY pgs=7199 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334015fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:06.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 secure :-1 s=READY pgs=7199 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f8334015fa0).stop 2021-03-09T10:11:06.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f8338074ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:06.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f8338074ed0 unknown :-1 s=CLOSED pgs=7199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:06.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:06.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338061ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:06.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:06.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ce40 con 0x7f8324003400 2021-03-09T10:11:06.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b450 con 0x7f8338015d30 2021-03-09T10:11:06.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d460 con 0x7f8324005e00 2021-03-09T10:11:06.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:11:06.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:06.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:11:06.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:06.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.183+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338061ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:06.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.187+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:06.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.187+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.187+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.187+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.187+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f8338015d30 2021-03-09T10:11:06.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.187+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338061ec0 secure :-1 s=READY pgs=7211 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:06.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.187+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:11:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.191+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:11:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:06.191+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338015d30 2021-03-09T10:11:06.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:11:06 smithi146 bash[13136]: debug 2021-03-09T10:11:06.185+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20730 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:11:11.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.187+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338061ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:11:11.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.187+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338061ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:11.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.187+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338061ec0 secure :-1 s=READY pgs=7211 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:11.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.187+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338061ec0 secure :-1 s=READY pgs=7211 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344049b10).stop 2021-03-09T10:11:11.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.187+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338061ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:11.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.187+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338061ec0 unknown :-1 s=CLOSED pgs=7211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:11.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.187+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:11.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.187+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:11.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:11.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324003400 2021-03-09T10:11:11.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012880 con 0x7f8338027b20 2021-03-09T10:11:11.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:11:11.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:11.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:11:11.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:11.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fed0 con 0x7f8324005e00 2021-03-09T10:11:11.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:11.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005f70 con 0x7f8338027b20 2021-03-09T10:11:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380735d0 secure :-1 s=READY pgs=7216 cs=0 l=1 rev1=1 rx=0x7f8334009bd0 tx=0x7f8334015fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833402da60 con 0x7f8338027b20 2021-03-09T10:11:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.191+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401f440 con 0x7f8338027b20 2021-03-09T10:11:11.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:11.195+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d220 con 0x7f8338027b20 2021-03-09T10:11:11.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:11:11 smithi146 bash[13136]: debug 2021-03-09T10:11:11.189+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20756 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:11:16.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f83380735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:11:16.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f83380735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:16.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380735d0 secure :-1 s=READY pgs=7216 cs=0 l=1 rev1=1 rx=0x7f8334009bd0 tx=0x7f8334015fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:16.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380735d0 secure :-1 s=READY pgs=7216 cs=0 l=1 rev1=1 rx=0x7f8334009bd0 tx=0x7f8334015fa0).stop 2021-03-09T10:11:16.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 msgr2=0x7f83380735d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:16.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338027b20 0x7f83380735d0 unknown :-1 s=CLOSED pgs=7216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:16.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338005750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:16.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:16.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:16.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:11:16.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012010 con 0x7f8338015d30 2021-03-09T10:11:16.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d460 con 0x7f8324005e00 2021-03-09T10:11:16.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:11:16.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:11:16.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:16.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:16.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.191+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338005750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:16.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.195+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:16.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.195+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:16.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.195+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:16.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.195+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:16.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.195+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024780 con 0x7f8338015d30 2021-03-09T10:11:16.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.195+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338005750 secure :-1 s=READY pgs=7225 cs=0 l=1 rev1=1 rx=0x7f833400bbf0 tx=0x7f8334015fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:16.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.195+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401cd50 con 0x7f8338015d30 2021-03-09T10:11:16.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.195+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404ba50 con 0x7f8338015d30 2021-03-09T10:11:16.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:16.199+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334019a80 con 0x7f8338015d30 2021-03-09T10:11:16.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:11:16 smithi146 bash[13136]: debug 2021-03-09T10:11:16.189+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20782 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:11:21.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338005750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:11:21.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338005750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:21.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338005750 secure :-1 s=READY pgs=7225 cs=0 l=1 rev1=1 rx=0x7f833400bbf0 tx=0x7f8334015fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:21.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338005750 secure :-1 s=READY pgs=7225 cs=0 l=1 rev1=1 rx=0x7f833400bbf0 tx=0x7f8334015fa0).stop 2021-03-09T10:11:21.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338005750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:21.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338005750 unknown :-1 s=CLOSED pgs=7225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:21.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:21.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:21.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:21.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011370 con 0x7f8324003400 2021-03-09T10:11:21.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8338006460 2021-03-09T10:11:21.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009710 con 0x7f8324005e00 2021-03-09T10:11:21.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:11:21.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:21.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:11:21.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:21.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.195+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:21.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.199+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:21.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:21.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.199+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:21.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:21.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.199+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800fe80 con 0x7f8338006460 2021-03-09T10:11:21.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 secure :-1 s=READY pgs=7233 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c007310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:21.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.199+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c061300 con 0x7f8338006460 2021-03-09T10:11:21.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.199+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c010860 con 0x7f8338006460 2021-03-09T10:11:21.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:21.203+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0252e0 con 0x7f8338006460 2021-03-09T10:11:21.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:11:21 smithi146 bash[13136]: debug 2021-03-09T10:11:21.197+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20808 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:11:26.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:11:26.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:26.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 secure :-1 s=READY pgs=7233 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c007310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:26.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 secure :-1 s=READY pgs=7233 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c007310).stop 2021-03-09T10:11:26.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f83380249b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:26.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 unknown :-1 s=CLOSED pgs=7233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:26.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:26.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:26.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:26.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069700 con 0x7f8324003400 2021-03-09T10:11:26.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007050 con 0x7f8338015d30 2021-03-09T10:11:26.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072690 con 0x7f8324005e00 2021-03-09T10:11:26.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:11:26.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:26.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:11:26.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:26.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.199+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:26.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.203+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:26.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.203+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:26.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.203+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:26.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.203+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:26.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.203+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380068e0 con 0x7f8338015d30 2021-03-09T10:11:26.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.203+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 secure :-1 s=READY pgs=7242 cs=0 l=1 rev1=1 rx=0x7f833802c830 tx=0x7f833c007310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:26.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.203+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c010860 con 0x7f8338015d30 2021-03-09T10:11:26.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c061300 con 0x7f8338015d30 2021-03-09T10:11:26.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:26.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0252e0 con 0x7f8338015d30 2021-03-09T10:11:26.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:11:26 smithi146 bash[13136]: debug 2021-03-09T10:11:26.201+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20834 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:11:31.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:11:31.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:31.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 secure :-1 s=READY pgs=7242 cs=0 l=1 rev1=1 rx=0x7f833802c830 tx=0x7f833c007310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:31.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 secure :-1 s=READY pgs=7242 cs=0 l=1 rev1=1 rx=0x7f833802c830 tx=0x7f833c007310).stop 2021-03-09T10:11:31.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:31.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 unknown :-1 s=CLOSED pgs=7242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:31.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:31.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:31.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:31.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b450 con 0x7f8324003400 2021-03-09T10:11:31.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d460 con 0x7f8338006460 2021-03-09T10:11:31.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d8f0 con 0x7f8324005e00 2021-03-09T10:11:31.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:11:31.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:11:31.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:31.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:31.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.203+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:31.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.207+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:31.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.207+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:31.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.207+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:31.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.207+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:31.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.207+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380628a0 con 0x7f8338006460 2021-03-09T10:11:31.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.207+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 secure :-1 s=READY pgs=7253 cs=0 l=1 rev1=1 rx=0x7f834404b940 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:31.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.211+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338006460 2021-03-09T10:11:31.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.211+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f8338006460 2021-03-09T10:11:31.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:31.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338006460 2021-03-09T10:11:31.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:11:31 smithi146 bash[13136]: debug 2021-03-09T10:11:31.201+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20860 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:11:36.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:11:36.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:36.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 secure :-1 s=READY pgs=7253 cs=0 l=1 rev1=1 rx=0x7f834404b940 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:36.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 secure :-1 s=READY pgs=7253 cs=0 l=1 rev1=1 rx=0x7f834404b940 tx=0x7f8344049b10).stop 2021-03-09T10:11:36.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f83380249b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:36.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 unknown :-1 s=CLOSED pgs=7253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:36.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:36.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:36.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:36.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380185f0 con 0x7f8324003400 2021-03-09T10:11:36.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007050 con 0x7f8338015d30 2021-03-09T10:11:36.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072690 con 0x7f8324005e00 2021-03-09T10:11:36.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:11:36.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:11:36.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:36.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:36.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.207+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:36.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.211+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:36.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.211+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:36.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.211+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:36.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.211+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:36.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.211+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801a650 con 0x7f8338015d30 2021-03-09T10:11:36.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.211+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 secure :-1 s=READY pgs=7262 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:36.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404ecb0 con 0x7f8338015d30 2021-03-09T10:11:36.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344017640 con 0x7f8338015d30 2021-03-09T10:11:36.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:36.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ac90 con 0x7f8338015d30 2021-03-09T10:11:36.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:11:36 smithi146 bash[13136]: debug 2021-03-09T10:11:36.209+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20885 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:11:41.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.211+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:11:41.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.211+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:41.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.211+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 secure :-1 s=READY pgs=7262 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:41.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.211+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 secure :-1 s=READY pgs=7262 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344049b10).stop 2021-03-09T10:11:41.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.211+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:41.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.211+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 unknown :-1 s=CLOSED pgs=7262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:41.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.211+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:41.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.211+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:11:41.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.211+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:41.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.211+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:41.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.211+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:41.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.211+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072b50 con 0x7f8324003400 2021-03-09T10:11:41.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.211+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012010 con 0x7f8338006460 2021-03-09T10:11:41.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.211+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d8f0 con 0x7f8324005e00 2021-03-09T10:11:41.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.215+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:41.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.215+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:11:41.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.215+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:41.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.219+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:41.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:41.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.219+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:41.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:41.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.219+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017d10 con 0x7f8338006460 2021-03-09T10:11:41.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 secure :-1 s=READY pgs=7272 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c007310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:41.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01d8e0 con 0x7f8338006460 2021-03-09T10:11:41.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01d3a0 con 0x7f8338006460 2021-03-09T10:11:41.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:41.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f8338006460 2021-03-09T10:11:41.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:11:41 smithi146 bash[13136]: debug 2021-03-09T10:11:41.213+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20908 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:11:46.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:11:46.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f83380249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:46.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 secure :-1 s=READY pgs=7272 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c007310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:46.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 secure :-1 s=READY pgs=7272 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c007310).stop 2021-03-09T10:11:46.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f83380249b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:46.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f83380249b0 unknown :-1 s=CLOSED pgs=7272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:46.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:46.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:46.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:46.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324003400 2021-03-09T10:11:46.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066c50 con 0x7f8338015d30 2021-03-09T10:11:46.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012880 con 0x7f8324005e00 2021-03-09T10:11:46.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:11:46.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:46.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:46.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:11:46.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.215+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:46.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.219+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:46.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.219+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:46.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.219+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:46.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.219+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:46.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.219+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007d80 con 0x7f8338015d30 2021-03-09T10:11:46.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.219+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 secure :-1 s=READY pgs=7277 cs=0 l=1 rev1=1 rx=0x7f8334000dd0 tx=0x7f8334015fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:46.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401ca50 con 0x7f8338015d30 2021-03-09T10:11:46.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340453c0 con 0x7f8338015d30 2021-03-09T10:11:46.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:46.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833400b980 con 0x7f8338015d30 2021-03-09T10:11:46.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:11:46 smithi146 bash[13136]: debug 2021-03-09T10:11:46.217+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20933 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:11:51.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.219+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:11:51.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.219+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:51.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.219+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 secure :-1 s=READY pgs=7277 cs=0 l=1 rev1=1 rx=0x7f8334000dd0 tx=0x7f8334015fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:51.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.219+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 secure :-1 s=READY pgs=7277 cs=0 l=1 rev1=1 rx=0x7f8334000dd0 tx=0x7f8334015fa0).stop 2021-03-09T10:11:51.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:51.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.219+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755e0 unknown :-1 s=CLOSED pgs=7277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:51.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.219+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:51.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.219+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338077ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:51.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.219+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:51.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:11:51.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380722f0 con 0x7f8338006460 2021-03-09T10:11:51.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072690 con 0x7f8324005e00 2021-03-09T10:11:51.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.219+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:11:51.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.219+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:51.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.223+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:11:51.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.223+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:51.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.223+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338077ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:51.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.223+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:51.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.223+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:51.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.223+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:51.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.223+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:51.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.223+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338016ba0 con 0x7f8338006460 2021-03-09T10:11:51.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.223+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338077ae0 secure :-1 s=READY pgs=7287 cs=0 l=1 rev1=1 rx=0x7f834404c690 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:51.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403cef0 con 0x7f8338006460 2021-03-09T10:11:51.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f8338006460 2021-03-09T10:11:51.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:51.227+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338006460 2021-03-09T10:11:51.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:11:51 smithi146 bash[13136]: debug 2021-03-09T10:11:51.216+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20959 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:11:56.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f8338077ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:11:56.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f8338077ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:56.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338077ae0 secure :-1 s=READY pgs=7287 cs=0 l=1 rev1=1 rx=0x7f834404c690 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:56.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338077ae0 secure :-1 s=READY pgs=7287 cs=0 l=1 rev1=1 rx=0x7f834404c690 tx=0x7f8344049b10).stop 2021-03-09T10:11:56.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f8338077ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:56.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338077ae0 unknown :-1 s=CLOSED pgs=7287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:56.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:56.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:56.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:56.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b450 con 0x7f8324003400 2021-03-09T10:11:56.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380722f0 con 0x7f8338015d30 2021-03-09T10:11:56.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380088b0 con 0x7f8324005e00 2021-03-09T10:11:56.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:11:56.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:56.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:11:56.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:56.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.222+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:56.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.226+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:56.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.226+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:56.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.226+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:56.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.226+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:56.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.226+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806d160 con 0x7f8338015d30 2021-03-09T10:11:56.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.226+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755c0 secure :-1 s=READY pgs=7294 cs=0 l=1 rev1=1 rx=0x7f8338072880 tx=0x7f833c007310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:56.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.226+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c06a6a0 con 0x7f8338015d30 2021-03-09T10:11:56.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.226+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0646e0 con 0x7f8338015d30 2021-03-09T10:11:56.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:11:56.230+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d1e0 con 0x7f8338015d30 2021-03-09T10:11:56.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:11:56 smithi146 bash[13136]: debug 2021-03-09T10:11:56.220+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 20985 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:12:01.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.226+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:12:01.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.226+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:01.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.226+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755c0 secure :-1 s=READY pgs=7294 cs=0 l=1 rev1=1 rx=0x7f8338072880 tx=0x7f833c007310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:01.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.226+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755c0 secure :-1 s=READY pgs=7294 cs=0 l=1 rev1=1 rx=0x7f8338072880 tx=0x7f833c007310).stop 2021-03-09T10:12:01.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.226+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:01.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.226+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755c0 unknown :-1 s=CLOSED pgs=7294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:01.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.226+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:01.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.230+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:01.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.230+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:01.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.230+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e9b0 con 0x7f8324003400 2021-03-09T10:12:01.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.230+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338006460 2021-03-09T10:12:01.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.230+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380088b0 con 0x7f8324005e00 2021-03-09T10:12:01.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.230+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:12:01.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.230+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:01.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.230+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:12:01.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.230+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:01.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.230+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:01.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.234+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:01.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.234+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:01.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.234+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:01.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.234+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:01.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.234+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338002ad0 con 0x7f8338006460 2021-03-09T10:12:01.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.234+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338021080 secure :-1 s=READY pgs=7308 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c007310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:01.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0188a0 con 0x7f8338006460 2021-03-09T10:12:01.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0188a0 con 0x7f8338006460 2021-03-09T10:12:01.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:01.238+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c036480 con 0x7f8338006460 2021-03-09T10:12:01.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:12:01 smithi146 bash[13136]: debug 2021-03-09T10:12:01.224+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21011 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:12:06.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:12:06.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:06.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338021080 secure :-1 s=READY pgs=7308 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c007310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:06.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338021080 secure :-1 s=READY pgs=7308 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c007310).stop 2021-03-09T10:12:06.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:06.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338021080 unknown :-1 s=CLOSED pgs=7308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:06.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:06.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:06.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:06.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019f00 con 0x7f8324003400 2021-03-09T10:12:06.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008700 con 0x7f8338015d30 2021-03-09T10:12:06.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012880 con 0x7f8324005e00 2021-03-09T10:12:06.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:12:06.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:06.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:12:06.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:06.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.230+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:06.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.234+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:06.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.234+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:06.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.234+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:06.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.234+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:06.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.234+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338016ba0 con 0x7f8338015d30 2021-03-09T10:12:06.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.234+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755c0 secure :-1 s=READY pgs=7311 cs=0 l=1 rev1=1 rx=0x7f833c054d30 tx=0x7f833c007310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:06.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c054780 con 0x7f8338015d30 2021-03-09T10:12:06.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c054b70 con 0x7f8338015d30 2021-03-09T10:12:06.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:06.238+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054310 con 0x7f8338015d30 2021-03-09T10:12:06.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:12:06 smithi146 bash[13136]: debug 2021-03-09T10:12:06.228+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21036 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:12:11.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:12:11.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:11.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755c0 secure :-1 s=READY pgs=7311 cs=0 l=1 rev1=1 rx=0x7f833c054d30 tx=0x7f833c007310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:11.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755c0 secure :-1 s=READY pgs=7311 cs=0 l=1 rev1=1 rx=0x7f833c054d30 tx=0x7f833c007310).stop 2021-03-09T10:12:11.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:11.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755c0 unknown :-1 s=CLOSED pgs=7311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:11.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f833801fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:11.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:11.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:11.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380722f0 con 0x7f8324003400 2021-03-09T10:12:11.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:12:11.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b450 con 0x7f8338006460 2021-03-09T10:12:11.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027260 con 0x7f8324005e00 2021-03-09T10:12:11.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f833801fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:11.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:12:11.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:11.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.234+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:11.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.238+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:11.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.238+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:11.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.238+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:11.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.238+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:11.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.238+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014430 con 0x7f8338006460 2021-03-09T10:12:11.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.238+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f833801fba0 secure :-1 s=READY pgs=7321 cs=0 l=1 rev1=1 rx=0x7f8334028480 tx=0x7f8334015fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:11.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.242+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8338006460 2021-03-09T10:12:11.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.242+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007c30 con 0x7f8338006460 2021-03-09T10:12:11.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:11.242+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340645a0 con 0x7f8338006460 2021-03-09T10:12:11.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:12:11 smithi146 bash[13136]: debug 2021-03-09T10:12:11.236+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21063 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:12:16.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f833801fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:12:16.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f833801fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:16.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f833801fba0 secure :-1 s=READY pgs=7321 cs=0 l=1 rev1=1 rx=0x7f8334028480 tx=0x7f8334015fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:16.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f833801fba0 secure :-1 s=READY pgs=7321 cs=0 l=1 rev1=1 rx=0x7f8334028480 tx=0x7f8334015fa0).stop 2021-03-09T10:12:16.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f833801fba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:16.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f833801fba0 unknown :-1 s=CLOSED pgs=7321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:16.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:16.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:16.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:16.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324003400 2021-03-09T10:12:16.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b450 con 0x7f8338015d30 2021-03-09T10:12:16.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380239d0 con 0x7f8324005e00 2021-03-09T10:12:16.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:12:16.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:16.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:12:16.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:16.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.238+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:16.246 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.242+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:16.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.242+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:16.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.242+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:16.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.242+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:16.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.242+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800fe80 con 0x7f8338015d30 2021-03-09T10:12:16.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.242+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7329 cs=0 l=1 rev1=1 rx=0x7f8334009b10 tx=0x7f8334015fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:16.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.246+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401b440 con 0x7f8338015d30 2021-03-09T10:12:16.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.246+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340646a0 con 0x7f8338015d30 2021-03-09T10:12:16.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:16.246+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334027250 con 0x7f8338015d30 2021-03-09T10:12:16.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:12:16 smithi146 bash[13136]: debug 2021-03-09T10:12:16.240+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21088 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:12:21.246 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.242+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:12:21.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.242+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:21.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.242+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7329 cs=0 l=1 rev1=1 rx=0x7f8334009b10 tx=0x7f8334015fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:21.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.242+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7329 cs=0 l=1 rev1=1 rx=0x7f8334009b10 tx=0x7f8334015fa0).stop 2021-03-09T10:12:21.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.242+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:21.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.242+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=CLOSED pgs=7329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:21.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.242+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:21.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.242+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338077aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:21.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.242+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:21.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.242+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008700 con 0x7f8324003400 2021-03-09T10:12:21.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.242+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009760 con 0x7f8338006460 2021-03-09T10:12:21.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.242+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:12:21.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.242+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:12:21.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.242+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:21.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.242+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:12:21.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.242+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:21.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.246+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338077aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:21.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.246+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:21.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.246+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:21.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.246+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:21.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.246+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:21.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.246+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338016ba0 con 0x7f8338006460 2021-03-09T10:12:21.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.246+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338077aa0 secure :-1 s=READY pgs=7338 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:21.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.250+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338006460 2021-03-09T10:12:21.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.250+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834403ba10 con 0x7f8338006460 2021-03-09T10:12:21.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:21.250+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338006460 2021-03-09T10:12:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:12:21 smithi146 bash[13136]: debug 2021-03-09T10:12:21.244+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21115 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:12:26.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.246+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f8338077aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:12:26.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.246+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f8338077aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:26.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.246+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338077aa0 secure :-1 s=READY pgs=7338 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:26.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.246+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338077aa0 secure :-1 s=READY pgs=7338 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344049b10).stop 2021-03-09T10:12:26.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.246+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 msgr2=0x7f8338077aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:26.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.246+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338006460 0x7f8338077aa0 unknown :-1 s=CLOSED pgs=7338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:26.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.246+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:26.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.246+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:26.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.246+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:26.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.246+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f8324003400 2021-03-09T10:12:26.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.246+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066230 con 0x7f8338015d30 2021-03-09T10:12:26.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.246+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009910 con 0x7f8324005e00 2021-03-09T10:12:26.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.246+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:12:26.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.246+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:26.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.246+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:26.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.250+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:12:26.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.250+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:26.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.250+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:26.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.250+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:26.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.250+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:26.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.250+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:26.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.250+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026400 con 0x7f8338015d30 2021-03-09T10:12:26.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.250+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7346 cs=0 l=1 rev1=1 rx=0x7f8334009bd0 tx=0x7f8334015fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:26.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.250+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334064900 con 0x7f8338015d30 2021-03-09T10:12:26.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.250+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403bed0 con 0x7f8338015d30 2021-03-09T10:12:26.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:26.254+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338015d30 2021-03-09T10:12:26.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:12:26 smithi146 bash[13136]: debug 2021-03-09T10:12:26.248+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21140 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:12:31.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:12:31.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:31.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7346 cs=0 l=1 rev1=1 rx=0x7f8334009bd0 tx=0x7f8334015fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:31.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7346 cs=0 l=1 rev1=1 rx=0x7f8334009bd0 tx=0x7f8334015fa0).stop 2021-03-09T10:12:31.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:31.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=CLOSED pgs=7346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:31.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338077e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:31.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:31.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:31.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:12:31.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023db0 con 0x7f8338075a90 2021-03-09T10:12:31.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009910 con 0x7f8324005e00 2021-03-09T10:12:31.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:12:31.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:31.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:12:31.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:31.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.250+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338077e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:31.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.254+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:31.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.254+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:31.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.254+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:31.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.254+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:31.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.254+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338066c50 con 0x7f8338075a90 2021-03-09T10:12:31.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.254+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338077e80 secure :-1 s=READY pgs=7355 cs=0 l=1 rev1=1 rx=0x7f8334058400 tx=0x7f8334015fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:31.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.254+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401aa90 con 0x7f8338075a90 2021-03-09T10:12:31.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.254+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401a860 con 0x7f8338075a90 2021-03-09T10:12:31.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:31.258+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334047250 con 0x7f8338075a90 2021-03-09T10:12:31.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:12:31 smithi146 bash[13136]: debug 2021-03-09T10:12:31.248+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21167 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:12:36.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338077e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:12:36.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338077e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:36.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338077e80 secure :-1 s=READY pgs=7355 cs=0 l=1 rev1=1 rx=0x7f8334058400 tx=0x7f8334015fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:36.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338077e80 secure :-1 s=READY pgs=7355 cs=0 l=1 rev1=1 rx=0x7f8334058400 tx=0x7f8334015fa0).stop 2021-03-09T10:12:36.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338077e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:36.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338077e80 unknown :-1 s=CLOSED pgs=7355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:36.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:36.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:36.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:36.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f8324003400 2021-03-09T10:12:36.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8338015d30 2021-03-09T10:12:36.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e230 con 0x7f8324005e00 2021-03-09T10:12:36.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:12:36.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:36.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:12:36.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:36.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:36.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:36.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:36.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:36.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:36.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.254+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800fe80 con 0x7f8338015d30 2021-03-09T10:12:36.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.258+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7363 cs=0 l=1 rev1=1 rx=0x7f8338020c80 tx=0x7f833c012450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:36.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.258+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018690 con 0x7f8338015d30 2021-03-09T10:12:36.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.258+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0187f0 con 0x7f8338015d30 2021-03-09T10:12:36.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:36.262+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c09a450 con 0x7f8338015d30 2021-03-09T10:12:36.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:12:36 smithi146 bash[13136]: debug 2021-03-09T10:12:36.252+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21192 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:12:41.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:12:41.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:41.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7363 cs=0 l=1 rev1=1 rx=0x7f8338020c80 tx=0x7f833c012450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:41.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7363 cs=0 l=1 rev1=1 rx=0x7f8338020c80 tx=0x7f833c012450).stop 2021-03-09T10:12:41.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:41.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=CLOSED pgs=7363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:41.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338016110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:41.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:41.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:41.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f8324003400 2021-03-09T10:12:41.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dce0 con 0x7f8338075a90 2021-03-09T10:12:41.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023a00 con 0x7f8324005e00 2021-03-09T10:12:41.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:12:41.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:41.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:12:41.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:41.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.257+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338016110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:41.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.261+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:41.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.261+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:41.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.261+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:41.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.261+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:41.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.261+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380628a0 con 0x7f8338075a90 2021-03-09T10:12:41.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.261+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338016110 secure :-1 s=READY pgs=7374 cs=0 l=1 rev1=1 rx=0x7f8334047040 tx=0x7f8334015fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:41.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334003790 con 0x7f8338075a90 2021-03-09T10:12:41.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833405f170 con 0x7f8338075a90 2021-03-09T10:12:41.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:41.265+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404bd80 con 0x7f8338075a90 2021-03-09T10:12:41.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:12:41 smithi146 bash[13136]: debug 2021-03-09T10:12:41.260+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21219 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:12:46.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338016110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:12:46.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338016110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:46.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338016110 secure :-1 s=READY pgs=7374 cs=0 l=1 rev1=1 rx=0x7f8334047040 tx=0x7f8334015fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:46.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338016110 secure :-1 s=READY pgs=7374 cs=0 l=1 rev1=1 rx=0x7f8334047040 tx=0x7f8334015fa0).stop 2021-03-09T10:12:46.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338016110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:46.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338016110 unknown :-1 s=CLOSED pgs=7374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:46.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:46.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:46.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:46.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023db0 con 0x7f8324003400 2021-03-09T10:12:46.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338015d30 2021-03-09T10:12:46.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:12:46.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d620 con 0x7f8324005e00 2021-03-09T10:12:46.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:46.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:12:46.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:46.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:46.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:46.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:46.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:46.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:46.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338061530 con 0x7f8338015d30 2021-03-09T10:12:46.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.261+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7381 cs=0 l=1 rev1=1 rx=0x7f833c036d80 tx=0x7f833c012450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:46.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.265+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c036ec0 con 0x7f8338015d30 2021-03-09T10:12:46.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.269+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338015d30 2021-03-09T10:12:46.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:46.269+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0458b0 con 0x7f8338015d30 2021-03-09T10:12:46.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:12:46 smithi146 bash[13136]: debug 2021-03-09T10:12:46.264+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21244 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:12:51.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:12:51.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:51.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7381 cs=0 l=1 rev1=1 rx=0x7f833c036d80 tx=0x7f833c012450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:51.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7381 cs=0 l=1 rev1=1 rx=0x7f833c036d80 tx=0x7f833c012450).stop 2021-03-09T10:12:51.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:51.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=CLOSED pgs=7381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:51.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338077e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:51.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:51.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:51.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dce0 con 0x7f8324003400 2021-03-09T10:12:51.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005e50 con 0x7f8338075a90 2021-03-09T10:12:51.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d620 con 0x7f8324005e00 2021-03-09T10:12:51.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:12:51.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:12:51.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:51.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:51.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.265+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338077e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:51.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.269+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:51.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.269+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:51.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.269+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:51.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.269+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:51.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.269+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800fe80 con 0x7f8338075a90 2021-03-09T10:12:51.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.269+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338077e80 secure :-1 s=READY pgs=7389 cs=0 l=1 rev1=1 rx=0x7f833403d040 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:51.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.269+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334027c20 con 0x7f8338075a90 2021-03-09T10:12:51.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.269+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340035c0 con 0x7f8338075a90 2021-03-09T10:12:51.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:51.273+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403bbd0 con 0x7f8338075a90 2021-03-09T10:12:51.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:12:51 smithi146 bash[13136]: debug 2021-03-09T10:12:51.268+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21271 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:12:56.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338077e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:12:56.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338077e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:56.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338077e80 secure :-1 s=READY pgs=7389 cs=0 l=1 rev1=1 rx=0x7f833403d040 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:56.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338077e80 secure :-1 s=READY pgs=7389 cs=0 l=1 rev1=1 rx=0x7f833403d040 tx=0x7f8334069000).stop 2021-03-09T10:12:56.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338077e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:56.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338077e80 unknown :-1 s=CLOSED pgs=7389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:56.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:56.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:56.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:56.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bbf0 con 0x7f8324003400 2021-03-09T10:12:56.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338015d30 2021-03-09T10:12:56.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d620 con 0x7f8324005e00 2021-03-09T10:12:56.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:12:56.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:12:56.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:56.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:56.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.269+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:56.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.273+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:56.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.273+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:56.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.273+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:56.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.273+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:56.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.273+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338021ab0 con 0x7f8338015d30 2021-03-09T10:12:56.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.273+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7398 cs=0 l=1 rev1=1 rx=0x7f833c01c4a0 tx=0x7f833c012450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:56.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.273+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c025930 con 0x7f8338015d30 2021-03-09T10:12:56.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.273+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c025d90 con 0x7f8338015d30 2021-03-09T10:12:56.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:12:56.277+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338015d30 2021-03-09T10:12:56.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:12:56 smithi146 bash[13136]: debug 2021-03-09T10:12:56.271+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21295 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:13:01.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:13:01.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:01.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7398 cs=0 l=1 rev1=1 rx=0x7f833c01c4a0 tx=0x7f833c012450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:01.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7398 cs=0 l=1 rev1=1 rx=0x7f833c01c4a0 tx=0x7f833c012450).stop 2021-03-09T10:13:01.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:01.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=CLOSED pgs=7398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:01.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338064e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:01.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:01.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:01.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023db0 con 0x7f8324003400 2021-03-09T10:13:01.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027260 con 0x7f8338075a90 2021-03-09T10:13:01.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007dd0 con 0x7f8324005e00 2021-03-09T10:13:01.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:13:01.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:01.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:13:01.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:01.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338064e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:01.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:01.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:01.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:01.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:01.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.273+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800fe80 con 0x7f8338075a90 2021-03-09T10:13:01.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.277+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338064e60 secure :-1 s=READY pgs=7407 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:01.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.277+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833403bed0 con 0x7f8338075a90 2021-03-09T10:13:01.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.277+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403bed0 con 0x7f8338075a90 2021-03-09T10:13:01.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:01.281+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334063180 con 0x7f8338075a90 2021-03-09T10:13:01.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:13:01 smithi146 bash[13136]: debug 2021-03-09T10:13:01.275+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21323 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:13:06.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338064e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:13:06.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338064e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:06.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338064e60 secure :-1 s=READY pgs=7407 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:06.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338064e60 secure :-1 s=READY pgs=7407 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).stop 2021-03-09T10:13:06.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338064e60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:06.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338064e60 unknown :-1 s=CLOSED pgs=7407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:06.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:06.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:06.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b1a0 con 0x7f8324003400 2021-03-09T10:13:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380170e0 con 0x7f8338015d30 2021-03-09T10:13:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:13:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005e50 con 0x7f8324005e00 2021-03-09T10:13:06.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:06.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:13:06.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:06.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:06.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:06.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:06.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:06.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:06.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8338015d30 2021-03-09T10:13:06.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.277+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7415 cs=0 l=1 rev1=1 rx=0x7f833400bbf0 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:06.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.281+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833406d180 con 0x7f8338015d30 2021-03-09T10:13:06.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.281+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334003e90 con 0x7f8338015d30 2021-03-09T10:13:06.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:06.285+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404ed40 con 0x7f8338015d30 2021-03-09T10:13:06.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:13:06 smithi146 bash[13136]: debug 2021-03-09T10:13:06.279+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21347 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:13:11.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:13:11.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:11.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7415 cs=0 l=1 rev1=1 rx=0x7f833400bbf0 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:11.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7415 cs=0 l=1 rev1=1 rx=0x7f833400bbf0 tx=0x7f8334069000).stop 2021-03-09T10:13:11.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:11.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=CLOSED pgs=7415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:11.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:11.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:11.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380720f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:11.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:13:11.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e230 con 0x7f8338075a90 2021-03-09T10:13:11.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a450 con 0x7f8324005e00 2021-03-09T10:13:11.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:13:11.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:11.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:13:11.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:11.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.281+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:11.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.285+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:11.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.285+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:11.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.285+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:11.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.285+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:11.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.285+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801de50 con 0x7f8338075a90 2021-03-09T10:13:11.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.285+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380720f0 secure :-1 s=READY pgs=7427 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c012450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:11.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.285+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04a3d0 con 0x7f8338075a90 2021-03-09T10:13:11.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.285+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338075a90 2021-03-09T10:13:11.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:11.293+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338075a90 2021-03-09T10:13:11.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:13:11 smithi146 bash[13136]: debug 2021-03-09T10:13:11.283+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21375 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:13:16.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f83380720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:13:16.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f83380720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:16.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380720f0 secure :-1 s=READY pgs=7427 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c012450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:16.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380720f0 secure :-1 s=READY pgs=7427 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c012450).stop 2021-03-09T10:13:16.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f83380720f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:16.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380720f0 unknown :-1 s=CLOSED pgs=7427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:16.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:16.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:16.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:16.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:13:16.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:13:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:13:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380208b0 con 0x7f8338015d30 2021-03-09T10:13:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066c00 con 0x7f8324005e00 2021-03-09T10:13:16.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:16.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:16.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:16.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:16.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:16.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.285+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014ad0 con 0x7f8338015d30 2021-03-09T10:13:16.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.289+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7433 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c012450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:16.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.289+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c09a310 con 0x7f8338015d30 2021-03-09T10:13:16.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.289+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c09a470 con 0x7f8338015d30 2021-03-09T10:13:16.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:16.293+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c2a0 con 0x7f8338015d30 2021-03-09T10:13:16.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:13:16 smithi146 bash[13136]: debug 2021-03-09T10:13:16.287+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21399 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:13:21.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:13:21.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:21.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7433 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c012450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:21.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 secure :-1 s=READY pgs=7433 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c012450).stop 2021-03-09T10:13:21.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f83380755a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:21.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f83380755a0 unknown :-1 s=CLOSED pgs=7433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:21.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380720f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:21.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:21.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:21.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380208b0 con 0x7f8324003400 2021-03-09T10:13:21.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8338075a90 2021-03-09T10:13:21.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:13:21.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:13:21.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:21.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:21.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:13:21.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:21.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:21.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:21.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:21.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:21.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.289+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024780 con 0x7f8338075a90 2021-03-09T10:13:21.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.293+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380720f0 secure :-1 s=READY pgs=7441 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:21.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.293+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334063e40 con 0x7f8338075a90 2021-03-09T10:13:21.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.293+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833405f170 con 0x7f8338075a90 2021-03-09T10:13:21.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:21.297+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401a360 con 0x7f8338075a90 2021-03-09T10:13:21.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:13:21 smithi146 bash[13136]: debug 2021-03-09T10:13:21.291+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21426 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:13:26.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.289+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f83380720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:13:26.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.289+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f83380720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:26.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.289+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380720f0 secure :-1 s=READY pgs=7441 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:26.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.289+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380720f0 secure :-1 s=READY pgs=7441 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).stop 2021-03-09T10:13:26.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f83380720f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:26.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380720f0 unknown :-1 s=CLOSED pgs=7441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:26.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:26.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:26.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338073fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:26.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:13:26.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:26.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:13:26.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:26.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011060 con 0x7f8324003400 2021-03-09T10:13:26.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d110 con 0x7f8338015d30 2021-03-09T10:13:26.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380208b0 con 0x7f8324005e00 2021-03-09T10:13:26.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338073fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:26.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:26.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:26.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:26.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:26.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.293+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a410 con 0x7f8338015d30 2021-03-09T10:13:26.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.297+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338073fb0 secure :-1 s=READY pgs=7450 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c012450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:26.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.297+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c061770 con 0x7f8338015d30 2021-03-09T10:13:26.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.297+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c03a000 con 0x7f8338015d30 2021-03-09T10:13:26.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:26.297+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d240 con 0x7f8338015d30 2021-03-09T10:13:26.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:13:26 smithi146 bash[13136]: debug 2021-03-09T10:13:26.295+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21451 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:13:31.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.292+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338073fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:13:31.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.292+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338073fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:31.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.292+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338073fb0 secure :-1 s=READY pgs=7450 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c012450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:31.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.292+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338073fb0 secure :-1 s=READY pgs=7450 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c012450).stop 2021-03-09T10:13:31.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338073fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:31.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338073fb0 unknown :-1 s=CLOSED pgs=7450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:31.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:31.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:31.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:13:31.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:31.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:13:31.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:31.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338075200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:31.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015640 con 0x7f8324003400 2021-03-09T10:13:31.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801abc0 con 0x7f8338075a90 2021-03-09T10:13:31.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:13:31.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338075200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:31.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:31.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:31.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:31.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:31.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.296+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028fe0 con 0x7f8338075a90 2021-03-09T10:13:31.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.300+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338075200 secure :-1 s=READY pgs=7461 cs=0 l=1 rev1=1 rx=0x7f833c02f930 tx=0x7f833c012450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:31.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.300+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c036340 con 0x7f8338075a90 2021-03-09T10:13:31.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.300+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f8338075a90 2021-03-09T10:13:31.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:31.304+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c036d50 con 0x7f8338075a90 2021-03-09T10:13:31.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:13:31 smithi146 bash[13136]: debug 2021-03-09T10:13:31.299+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21478 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:13:36.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338075200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:13:36.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338075200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:36.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338075200 secure :-1 s=READY pgs=7461 cs=0 l=1 rev1=1 rx=0x7f833c02f930 tx=0x7f833c012450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:36.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338075200 secure :-1 s=READY pgs=7461 cs=0 l=1 rev1=1 rx=0x7f833c02f930 tx=0x7f833c012450).stop 2021-03-09T10:13:36.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338075200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:36.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338075200 unknown :-1 s=CLOSED pgs=7461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:36.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338072800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:36.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:36.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:13:36.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:36.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:36.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cad0 con 0x7f8324003400 2021-03-09T10:13:36.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021d00 con 0x7f8338015d30 2021-03-09T10:13:36.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324005e00 2021-03-09T10:13:36.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338072800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:36.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:13:36.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:36.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:36.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:36.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:36.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:36.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801de50 con 0x7f8338015d30 2021-03-09T10:13:36.309 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.300+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338072800 secure :-1 s=READY pgs=7467 cs=0 l=1 rev1=1 rx=0x7f8334004f90 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:36.309 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.304+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334027250 con 0x7f8338015d30 2021-03-09T10:13:36.309 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.304+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340273b0 con 0x7f8338015d30 2021-03-09T10:13:36.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:36.308+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404ed20 con 0x7f8338015d30 2021-03-09T10:13:36.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:13:36 smithi146 bash[13136]: debug 2021-03-09T10:13:36.303+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21503 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:13:41.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.300+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338072800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:13:41.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.300+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338072800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:41.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.300+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338072800 secure :-1 s=READY pgs=7467 cs=0 l=1 rev1=1 rx=0x7f8334004f90 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:41.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.300+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338072800 secure :-1 s=READY pgs=7467 cs=0 l=1 rev1=1 rx=0x7f8334004f90 tx=0x7f8334069000).stop 2021-03-09T10:13:41.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.300+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338072800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:41.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.300+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338072800 unknown :-1 s=CLOSED pgs=7467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:41.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.300+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:41.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.300+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833800dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:41.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.304+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:41.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.304+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016350 con 0x7f8324003400 2021-03-09T10:13:41.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.304+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015640 con 0x7f8338075a90 2021-03-09T10:13:41.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.304+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cad0 con 0x7f8324005e00 2021-03-09T10:13:41.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.304+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833800dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:41.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.304+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:41.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.304+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:41.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.304+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:41.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.304+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:41.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.304+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024780 con 0x7f8338075a90 2021-03-09T10:13:41.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.304+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833800dc70 secure :-1 s=READY pgs=7476 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:41.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.308+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338075a90 2021-03-09T10:13:41.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.308+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f8338075a90 2021-03-09T10:13:41.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:41.312+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338075a90 2021-03-09T10:13:41.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:13:41 smithi146 bash[13136]: debug 2021-03-09T10:13:41.307+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21530 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:13:46.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f833800dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:13:46.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f833800dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:46.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833800dc70 secure :-1 s=READY pgs=7476 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:46.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833800dc70 secure :-1 s=READY pgs=7476 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344049b10).stop 2021-03-09T10:13:46.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f833800dc70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:46.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833800dc70 unknown :-1 s=CLOSED pgs=7476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:46.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:46.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:46.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:46.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a5a0 con 0x7f8324003400 2021-03-09T10:13:46.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008a60 con 0x7f8338015d30 2021-03-09T10:13:46.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:13:46.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:13:46.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:13:46.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:46.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:46.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:46.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:46.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:46.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:46.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:46.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801f100 con 0x7f8338015d30 2021-03-09T10:13:46.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.308+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a800 secure :-1 s=READY pgs=7485 cs=0 l=1 rev1=1 rx=0x7f833403afb0 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:46.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.312+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401a420 con 0x7f8338015d30 2021-03-09T10:13:46.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.312+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403bf00 con 0x7f8338015d30 2021-03-09T10:13:46.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:46.316+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334007b30 con 0x7f8338015d30 2021-03-09T10:13:46.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:13:46 smithi146 bash[13136]: debug 2021-03-09T10:13:46.311+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21555 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:13:51.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.308+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:13:51.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.308+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.308+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a800 secure :-1 s=READY pgs=7485 cs=0 l=1 rev1=1 rx=0x7f833403afb0 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.308+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a800 secure :-1 s=READY pgs=7485 cs=0 l=1 rev1=1 rx=0x7f833403afb0 tx=0x7f8334069000).stop 2021-03-09T10:13:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.308+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833801a800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.308+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833801a800 unknown :-1 s=CLOSED pgs=7485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.308+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.308+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338073fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.308+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.308+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e700 con 0x7f8324003400 2021-03-09T10:13:51.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.308+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027450 con 0x7f8338075a90 2021-03-09T10:13:51.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.308+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380267b0 con 0x7f8324005e00 2021-03-09T10:13:51.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.312+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:13:51.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.312+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:51.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.312+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:13:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.312+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.312+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338073fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.312+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.312+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.312+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.312+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.312+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338066200 con 0x7f8338075a90 2021-03-09T10:13:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.312+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338073fb0 secure :-1 s=READY pgs=7496 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.316+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338075a90 2021-03-09T10:13:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.316+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f8338075a90 2021-03-09T10:13:51.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:51.320+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440791e0 con 0x7f8338075a90 2021-03-09T10:13:51.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:13:51 smithi146 bash[13136]: debug 2021-03-09T10:13:51.315+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21581 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:13:56.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.312+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338073fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:13:56.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.312+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338073fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:56.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.312+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338073fb0 secure :-1 s=READY pgs=7496 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:56.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.312+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338073fb0 secure :-1 s=READY pgs=7496 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344049b10).stop 2021-03-09T10:13:56.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.312+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f8338073fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:56.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.312+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f8338073fb0 unknown :-1 s=CLOSED pgs=7496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:56.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.312+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:56.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.312+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:56.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.312+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338065c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:56.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.312+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014b50 con 0x7f8324003400 2021-03-09T10:13:56.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.312+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020750 con 0x7f8338015d30 2021-03-09T10:13:56.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.312+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d6f0 con 0x7f8324005e00 2021-03-09T10:13:56.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.312+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:13:56.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.312+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:56.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.316+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:13:56.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.316+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:56.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.316+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338065c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:56.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.316+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:56.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.316+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:56.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.316+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:56.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.316+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:56.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.316+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028fe0 con 0x7f8338015d30 2021-03-09T10:13:56.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.316+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338065c60 secure :-1 s=READY pgs=7502 cs=0 l=1 rev1=1 rx=0x7f833c072ff0 tx=0x7f833c012450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:56.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.320+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c054ce0 con 0x7f8338015d30 2021-03-09T10:13:56.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.320+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c09a310 con 0x7f8338015d30 2021-03-09T10:13:56.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:13:56.320+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f8338015d30 2021-03-09T10:13:56.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:13:56 smithi146 bash[13136]: debug 2021-03-09T10:13:56.318+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21607 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:14:01.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338065c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:14:01.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338065c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:01.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338065c60 secure :-1 s=READY pgs=7502 cs=0 l=1 rev1=1 rx=0x7f833c072ff0 tx=0x7f833c012450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:01.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338065c60 secure :-1 s=READY pgs=7502 cs=0 l=1 rev1=1 rx=0x7f833c072ff0 tx=0x7f833c012450).stop 2021-03-09T10:14:01.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338065c60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:01.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338065c60 unknown :-1 s=CLOSED pgs=7502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:01.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:01.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833802d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:01.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:01.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023db0 con 0x7f8324003400 2021-03-09T10:14:01.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009ac0 con 0x7f8338075a90 2021-03-09T10:14:01.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e700 con 0x7f8324005e00 2021-03-09T10:14:01.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:14:01.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:01.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833802d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:01.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:14:01.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.320+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:01.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.324+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:01.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.324+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:01.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.324+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:01.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.324+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:01.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.324+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024780 con 0x7f8338075a90 2021-03-09T10:14:01.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.324+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833802d4e0 secure :-1 s=READY pgs=7512 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:01.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.324+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f8338075a90 2021-03-09T10:14:01.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.324+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338075a90 2021-03-09T10:14:01.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:01.328+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338075a90 2021-03-09T10:14:01.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:14:01 smithi146 bash[13136]: debug 2021-03-09T10:14:01.322+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21632 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:14:06.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f833802d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:14:06.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f833802d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:06.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833802d4e0 secure :-1 s=READY pgs=7512 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:06.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833802d4e0 secure :-1 s=READY pgs=7512 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344049b10).stop 2021-03-09T10:14:06.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f833802d4e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:06.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833802d4e0 unknown :-1 s=CLOSED pgs=7512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:06.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:06.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:06.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:06.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066bd0 con 0x7f8324003400 2021-03-09T10:14:06.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028970 con 0x7f8338015d30 2021-03-09T10:14:06.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:14:06.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:14:06.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:06.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:06.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:14:06.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:06.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:06.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:06.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:06.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:06.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028fe0 con 0x7f8338015d30 2021-03-09T10:14:06.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d9d0 secure :-1 s=READY pgs=7519 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:06.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334049c30 con 0x7f8338015d30 2021-03-09T10:14:06.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.324+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007c30 con 0x7f8338015d30 2021-03-09T10:14:06.333 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:06.328+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334033490 con 0x7f8338015d30 2021-03-09T10:14:06.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:14:06 smithi146 bash[13136]: debug 2021-03-09T10:14:06.326+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21655 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:14:11.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:14:11.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:11.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d9d0 secure :-1 s=READY pgs=7519 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:11.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d9d0 secure :-1 s=READY pgs=7519 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).stop 2021-03-09T10:14:11.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802d9d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:11.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d9d0 unknown :-1 s=CLOSED pgs=7519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:11.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:11.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:11.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:11.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e8b0 con 0x7f8324003400 2021-03-09T10:14:11.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009ac0 con 0x7f8338075a90 2021-03-09T10:14:11.333 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d6f0 con 0x7f8324005e00 2021-03-09T10:14:11.333 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:14:11.333 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:11.333 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:14:11.334 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:11.334 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:11.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:11.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:11.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:11.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:11.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800e150 con 0x7f8338075a90 2021-03-09T10:14:11.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 secure :-1 s=READY pgs=7529 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:11.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.328+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340079c0 con 0x7f8338075a90 2021-03-09T10:14:11.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.332+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401f440 con 0x7f8338075a90 2021-03-09T10:14:11.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:11.336+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334033490 con 0x7f8338075a90 2021-03-09T10:14:11.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:14:11 smithi146 bash[13136]: debug 2021-03-09T10:14:11.330+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21680 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:14:16.334 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.328+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f83380145f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:14:16.334 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.328+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f83380145f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:16.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.328+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 secure :-1 s=READY pgs=7529 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:16.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.328+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 secure :-1 s=READY pgs=7529 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).stop 2021-03-09T10:14:16.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.328+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f83380145f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:16.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.328+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 unknown :-1 s=CLOSED pgs=7529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:16.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.328+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:16.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.328+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:16.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.328+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:16.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.328+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:14:16.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.328+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:14:16.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.328+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011060 con 0x7f8338015d30 2021-03-09T10:14:16.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.328+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066bd0 con 0x7f8324005e00 2021-03-09T10:14:16.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.328+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:16.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.328+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:14:16.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.328+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:16.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.332+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:16.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.332+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:16.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.332+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:16.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.332+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:16.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.332+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:16.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.332+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ed40 con 0x7f8338015d30 2021-03-09T10:14:16.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.332+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063e40 secure :-1 s=READY pgs=7537 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:16.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.336+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:14:16.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.336+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f8338015d30 2021-03-09T10:14:16.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:16.336+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338015d30 2021-03-09T10:14:16.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:14:16 smithi146 bash[13136]: debug 2021-03-09T10:14:16.334+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21707 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:14:21.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338063e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:14:21.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338063e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:21.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063e40 secure :-1 s=READY pgs=7537 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:21.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063e40 secure :-1 s=READY pgs=7537 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344049b10).stop 2021-03-09T10:14:21.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338063e40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:21.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338063e40 unknown :-1 s=CLOSED pgs=7537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:21.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:21.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:21.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:21.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380109d0 con 0x7f8324003400 2021-03-09T10:14:21.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020750 con 0x7f8338075a90 2021-03-09T10:14:21.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028970 con 0x7f8324005e00 2021-03-09T10:14:21.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:14:21.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:21.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:14:21.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:21.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:21.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:21.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:21.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:21.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:21.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028fe0 con 0x7f8338075a90 2021-03-09T10:14:21.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.335+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 secure :-1 s=READY pgs=7545 cs=0 l=1 rev1=1 rx=0x7f833c00fd40 tx=0x7f833c012450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:21.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.339+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f8338075a90 2021-03-09T10:14:21.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.339+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c036550 con 0x7f8338075a90 2021-03-09T10:14:21.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:21.339+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338075a90 2021-03-09T10:14:21.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:14:21 smithi146 bash[13136]: debug 2021-03-09T10:14:21.338+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21732 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:14:26.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.339+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f83380145f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:14:26.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.339+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f83380145f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:26.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.339+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 secure :-1 s=READY pgs=7545 cs=0 l=1 rev1=1 rx=0x7f833c00fd40 tx=0x7f833c012450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:26.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.339+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 secure :-1 s=READY pgs=7545 cs=0 l=1 rev1=1 rx=0x7f833c00fd40 tx=0x7f833c012450).stop 2021-03-09T10:14:26.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.339+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f83380145f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:26.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.339+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 unknown :-1 s=CLOSED pgs=7545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:26.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.339+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:26.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.339+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:26.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.339+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:26.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.339+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b40 con 0x7f8324003400 2021-03-09T10:14:26.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.339+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011060 con 0x7f8338015d30 2021-03-09T10:14:26.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.339+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066bd0 con 0x7f8324005e00 2021-03-09T10:14:26.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.339+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:14:26.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.339+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:26.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.343+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:14:26.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.343+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:26.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.343+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:26.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.343+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:26.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.343+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:26.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.343+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:26.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.343+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:26.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.343+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8338015d30 2021-03-09T10:14:26.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.343+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d4e0 secure :-1 s=READY pgs=7554 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:26.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.343+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:14:26.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.347+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344014a00 con 0x7f8338015d30 2021-03-09T10:14:26.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:26.347+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440791e0 con 0x7f8338015d30 2021-03-09T10:14:26.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:14:26 smithi146 bash[13136]: debug 2021-03-09T10:14:26.342+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21759 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:14:31.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.343+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:14:31.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.343+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:31.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.343+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d4e0 secure :-1 s=READY pgs=7554 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:31.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.343+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d4e0 secure :-1 s=READY pgs=7554 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344049b10).stop 2021-03-09T10:14:31.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.343+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802d4e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:31.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.343+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d4e0 unknown :-1 s=CLOSED pgs=7554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:31.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.343+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:31.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.343+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:31.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.343+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:31.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.343+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028970 con 0x7f8324003400 2021-03-09T10:14:31.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.343+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8338075a90 2021-03-09T10:14:31.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.343+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324005e00 2021-03-09T10:14:31.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.343+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:14:31.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.343+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:31.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.343+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:14:31.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.343+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:31.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.347+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:31.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.347+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:31.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.347+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:31.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.347+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:31.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.347+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:31.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.347+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338021b50 con 0x7f8338075a90 2021-03-09T10:14:31.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.347+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 secure :-1 s=READY pgs=7563 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c012450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:31.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.347+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0183b0 con 0x7f8338075a90 2021-03-09T10:14:31.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.347+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338075a90 2021-03-09T10:14:31.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:31.351+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004150 con 0x7f8338075a90 2021-03-09T10:14:31.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:14:31 smithi146 bash[13136]: debug 2021-03-09T10:14:31.346+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21784 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:14:36.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.347+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f83380145f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:14:36.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.347+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f83380145f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:36.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.347+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 secure :-1 s=READY pgs=7563 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c012450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:36.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.347+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 secure :-1 s=READY pgs=7563 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c012450).stop 2021-03-09T10:14:36.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.347+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f83380145f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:36.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.347+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f83380145f0 unknown :-1 s=CLOSED pgs=7563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:36.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.347+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:36.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.347+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338008730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:36.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.347+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:36.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.347+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016e10 con 0x7f8324003400 2021-03-09T10:14:36.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.347+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074390 con 0x7f8338015d30 2021-03-09T10:14:36.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.347+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:14:36.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.347+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:14:36.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.347+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:36.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.351+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:14:36.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.351+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:36.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.351+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:36.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.351+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:36.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.351+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:36.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.351+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:36.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.351+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:36.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.351+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800fe80 con 0x7f8338015d30 2021-03-09T10:14:36.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.351+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338008730 secure :-1 s=READY pgs=7572 cs=0 l=1 rev1=1 rx=0x7f8344075290 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:36.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.355+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344057f10 con 0x7f8338015d30 2021-03-09T10:14:36.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.355+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019e90 con 0x7f8338015d30 2021-03-09T10:14:36.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:36.359+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338015d30 2021-03-09T10:14:36.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:14:36 smithi146 bash[13136]: debug 2021-03-09T10:14:36.350+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21811 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:14:41.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.351+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338008730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:14:41.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.351+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338008730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:41.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.351+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338008730 secure :-1 s=READY pgs=7572 cs=0 l=1 rev1=1 rx=0x7f8344075290 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:41.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.351+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338008730 secure :-1 s=READY pgs=7572 cs=0 l=1 rev1=1 rx=0x7f8344075290 tx=0x7f8344049b10).stop 2021-03-09T10:14:41.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.351+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:41.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.351+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338008730 unknown :-1 s=CLOSED pgs=7572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:41.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.351+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:41.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.351+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833801e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:41.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.351+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:41.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.351+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f8324003400 2021-03-09T10:14:41.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.351+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8338075a90 2021-03-09T10:14:41.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.351+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020750 con 0x7f8324005e00 2021-03-09T10:14:41.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.351+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:14:41.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.351+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:41.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.355+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:14:41.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.355+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:41.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.355+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833801e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:41.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.355+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:41.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.355+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:41.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.355+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:41.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.355+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:41.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.355+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ed40 con 0x7f8338075a90 2021-03-09T10:14:41.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.359+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833801e1a0 secure :-1 s=READY pgs=7585 cs=0 l=1 rev1=1 rx=0x7f834402bfe0 tx=0x7f8344049b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:41.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.359+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405eee0 con 0x7f8338075a90 2021-03-09T10:14:41.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.363+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344017640 con 0x7f8338075a90 2021-03-09T10:14:41.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:41.363+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ac90 con 0x7f8338075a90 2021-03-09T10:14:41.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:14:41 smithi146 bash[13136]: debug 2021-03-09T10:14:41.354+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21837 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:14:46.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.355+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f833801e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:14:46.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.355+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f833801e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:46.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.355+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833801e1a0 secure :-1 s=READY pgs=7585 cs=0 l=1 rev1=1 rx=0x7f834402bfe0 tx=0x7f8344049b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:46.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.355+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833801e1a0 secure :-1 s=READY pgs=7585 cs=0 l=1 rev1=1 rx=0x7f834402bfe0 tx=0x7f8344049b10).stop 2021-03-09T10:14:46.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.355+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f833801e1a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:46.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.355+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833801e1a0 unknown :-1 s=CLOSED pgs=7585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:46.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.355+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:46.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.355+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:46.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.355+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:46.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.359+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067670 con 0x7f8324003400 2021-03-09T10:14:46.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.359+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028970 con 0x7f8338015d30 2021-03-09T10:14:46.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.359+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:46.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.359+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:14:46.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.359+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:46.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.359+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806daa0 con 0x7f8324005e00 2021-03-09T10:14:46.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.359+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:14:46.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.359+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:46.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.359+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:46.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.359+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:46.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.359+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:46.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.359+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:46.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.359+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802d2c0 con 0x7f8338015d30 2021-03-09T10:14:46.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.359+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d4e0 secure :-1 s=READY pgs=7589 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:46.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.363+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833402db30 con 0x7f8338015d30 2021-03-09T10:14:46.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.363+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338015d30 2021-03-09T10:14:46.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:46.367+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833406d180 con 0x7f8338015d30 2021-03-09T10:14:46.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:14:46 smithi146 bash[13136]: debug 2021-03-09T10:14:46.358+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21863 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:14:50.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:50.643+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2021-03-09T10:14:50.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:50.643+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:50.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:50.643+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f833c018660 tx=0x7f833c039000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:50.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:50.643+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f833c018660 tx=0x7f833c039000).stop 2021-03-09T10:14:50.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:50.643+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T10:14:50.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:50.643+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:50.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:50.643+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:50.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:50.643+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338075a90 0x7f8338023900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:50.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:50.643+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f834400de80 con 0x7f8338075a90 2021-03-09T10:14:50.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:50.643+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338075a90 0x7f8338023900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:50.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:50.643+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338075a90 0x7f8338023900 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c06e000).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:51.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.359+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:14:51.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.359+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:51.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.359+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d4e0 secure :-1 s=READY pgs=7589 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:51.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.359+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d4e0 secure :-1 s=READY pgs=7589 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334069000).stop 2021-03-09T10:14:51.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833802d4e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:51.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833802d4e0 unknown :-1 s=CLOSED pgs=7589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:51.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:51.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:51.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:51.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324003400 2021-03-09T10:14:51.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d6f0 con 0x7f8338071d10 2021-03-09T10:14:51.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:14:51.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:14:51.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:51.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:14:51.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:51.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:51.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:51.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:51.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:51.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:51.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.363+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8338071d10 2021-03-09T10:14:51.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.367+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7598 cs=0 l=1 rev1=1 rx=0x7f8334004f90 tx=0x7f8334069000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:51.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.367+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404b970 con 0x7f8338071d10 2021-03-09T10:14:51.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.367+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402aea0 con 0x7f8338071d10 2021-03-09T10:14:51.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:51.371+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401a8f0 con 0x7f8338071d10 2021-03-09T10:14:51.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:14:51 smithi146 bash[13136]: debug 2021-03-09T10:14:51.362+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21889 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:14:56.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:14:56.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:56.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7598 cs=0 l=1 rev1=1 rx=0x7f8334004f90 tx=0x7f8334069000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:56.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7598 cs=0 l=1 rev1=1 rx=0x7f8334004f90 tx=0x7f8334069000).stop 2021-03-09T10:14:56.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:56.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:56.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338004e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:56.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:56.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:56.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806daa0 con 0x7f8324003400 2021-03-09T10:14:56.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028970 con 0x7f8338015d30 2021-03-09T10:14:56.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b40 con 0x7f8324005e00 2021-03-09T10:14:56.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:14:56.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:56.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:14:56.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:56.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338004e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:56.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:56.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:56.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:56.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:56.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338021b50 con 0x7f8338015d30 2021-03-09T10:14:56.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.367+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338004e30 secure :-1 s=READY pgs=7606 cs=0 l=1 rev1=1 rx=0x7f833406d6d0 tx=0x7f8334032000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:56.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.371+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404b970 con 0x7f8338015d30 2021-03-09T10:14:56.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.371+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401a8f0 con 0x7f8338015d30 2021-03-09T10:14:56.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:14:56.371+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404ed80 con 0x7f8338015d30 2021-03-09T10:14:56.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:14:56 smithi146 bash[13136]: debug 2021-03-09T10:14:56.366+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21915 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:15:01.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.367+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338004e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:15:01.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.367+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338004e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:01.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.367+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338004e30 secure :-1 s=READY pgs=7606 cs=0 l=1 rev1=1 rx=0x7f833406d6d0 tx=0x7f8334032000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:01.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.367+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338004e30 secure :-1 s=READY pgs=7606 cs=0 l=1 rev1=1 rx=0x7f833406d6d0 tx=0x7f8334032000).stop 2021-03-09T10:15:01.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.367+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338004e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:01.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.367+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338004e30 unknown :-1 s=CLOSED pgs=7606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:01.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.367+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:01.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.367+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:01.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.367+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:01.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.367+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:15:01.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.367+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806daa0 con 0x7f8338071d10 2021-03-09T10:15:01.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.367+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011260 con 0x7f8324005e00 2021-03-09T10:15:01.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.367+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:15:01.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.371+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:01.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.371+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:15:01.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.371+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:01.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.371+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:01.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.371+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:01.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.371+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:01.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.371+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:01.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.371+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:01.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.371+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ed40 con 0x7f8338071d10 2021-03-09T10:15:01.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.371+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7615 cs=0 l=1 rev1=1 rx=0x7f83340034e0 tx=0x7f8334032000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:01.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.375+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334023380 con 0x7f8338071d10 2021-03-09T10:15:01.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.375+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334003550 con 0x7f8338071d10 2021-03-09T10:15:01.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:01.379+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404eca0 con 0x7f8338071d10 2021-03-09T10:15:01.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:15:01 smithi146 bash[13136]: debug 2021-03-09T10:15:01.369+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21941 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:15:06.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.371+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:15:06.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.371+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:06.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.371+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7615 cs=0 l=1 rev1=1 rx=0x7f83340034e0 tx=0x7f8334032000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:06.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.371+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7615 cs=0 l=1 rev1=1 rx=0x7f83340034e0 tx=0x7f8334032000).stop 2021-03-09T10:15:06.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.371+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:06.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.371+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:06.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.375+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:06.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.375+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338005850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:06.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.375+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:06.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.375+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b40 con 0x7f8324003400 2021-03-09T10:15:06.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.375+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806daa0 con 0x7f8338015d30 2021-03-09T10:15:06.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.375+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d6f0 con 0x7f8324005e00 2021-03-09T10:15:06.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.375+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:15:06.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.375+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:06.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.375+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:15:06.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.375+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:06.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.375+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338005850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:06.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.375+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:06.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.375+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:06.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.375+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:06.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.375+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:06.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.375+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ed40 con 0x7f8338015d30 2021-03-09T10:15:06.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.379+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338005850 secure :-1 s=READY pgs=7625 cs=0 l=1 rev1=1 rx=0x7f834405e970 tx=0x7f834400e1c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:06.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.379+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338015d30 2021-03-09T10:15:06.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338015d30 2021-03-09T10:15:06.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:06.383+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338015d30 2021-03-09T10:15:06.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:15:06 smithi146 bash[13136]: debug 2021-03-09T10:15:06.373+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21967 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:15:11.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338005850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:15:11.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338005850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:11.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338005850 secure :-1 s=READY pgs=7625 cs=0 l=1 rev1=1 rx=0x7f834405e970 tx=0x7f834400e1c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:11.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338005850 secure :-1 s=READY pgs=7625 cs=0 l=1 rev1=1 rx=0x7f834405e970 tx=0x7f834400e1c0).stop 2021-03-09T10:15:11.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338005850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:11.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338005850 unknown :-1 s=CLOSED pgs=7625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:11.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:11.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:11.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:11.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801adc0 con 0x7f8324003400 2021-03-09T10:15:11.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806daa0 con 0x7f8338071d10 2021-03-09T10:15:11.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073b70 con 0x7f8324005e00 2021-03-09T10:15:11.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:15:11.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:11.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:11.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:15:11.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:11.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:11.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:11.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:11.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:11.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338073370 con 0x7f8338071d10 2021-03-09T10:15:11.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.378+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7632 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c098000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:11.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.382+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c018280 con 0x7f8338071d10 2021-03-09T10:15:11.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.382+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f8338071d10 2021-03-09T10:15:11.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:11.386+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338071d10 2021-03-09T10:15:11.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:15:11 smithi146 bash[13136]: debug 2021-03-09T10:15:11.377+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 21993 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:15:16.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:15:16.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:16.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7632 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c098000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:16.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7632 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c098000).stop 2021-03-09T10:15:16.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:16.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:16.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:16.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:16.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:16.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020a60 con 0x7f8324003400 2021-03-09T10:15:16.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b40 con 0x7f8338015d30 2021-03-09T10:15:16.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015b90 con 0x7f8324005e00 2021-03-09T10:15:16.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:15:16.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:16.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:15:16.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:16.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:16.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:16.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:16.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:16.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:16.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338025480 con 0x7f8338015d30 2021-03-09T10:15:16.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.382+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800a0c0 secure :-1 s=READY pgs=7641 cs=0 l=1 rev1=1 rx=0x7f833400bbf0 tx=0x7f8334032000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:16.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.386+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404ead0 con 0x7f8338015d30 2021-03-09T10:15:16.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.386+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334062e30 con 0x7f8338015d30 2021-03-09T10:15:16.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:16.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404eca0 con 0x7f8338015d30 2021-03-09T10:15:16.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:15:16 smithi146 bash[13136]: debug 2021-03-09T10:15:16.381+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22018 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:15:21.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:15:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800a0c0 secure :-1 s=READY pgs=7641 cs=0 l=1 rev1=1 rx=0x7f833400bbf0 tx=0x7f8334032000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:21.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800a0c0 secure :-1 s=READY pgs=7641 cs=0 l=1 rev1=1 rx=0x7f833400bbf0 tx=0x7f8334032000).stop 2021-03-09T10:15:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f833800a0c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f833800a0c0 unknown :-1 s=CLOSED pgs=7641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:21.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:21.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073b70 con 0x7f8324003400 2021-03-09T10:15:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d6f0 con 0x7f8338071d10 2021-03-09T10:15:21.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801adc0 con 0x7f8324005e00 2021-03-09T10:15:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:15:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:15:21.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:21.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:21.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.386+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338021b50 con 0x7f8338071d10 2021-03-09T10:15:21.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.390+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7649 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f8334032000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:21.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340183e0 con 0x7f8338071d10 2021-03-09T10:15:21.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403bc60 con 0x7f8338071d10 2021-03-09T10:15:21.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:21.394+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401a8f0 con 0x7f8338071d10 2021-03-09T10:15:21.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:15:21 smithi146 bash[13136]: debug 2021-03-09T10:15:21.385+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22045 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:15:26.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:15:26.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:26.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7649 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f8334032000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:26.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7649 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f8334032000).stop 2021-03-09T10:15:26.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:26.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:26.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:26.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:26.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:26.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:26.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:15:26.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:26.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801adc0 con 0x7f8324003400 2021-03-09T10:15:26.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d8f0 con 0x7f8338015d30 2021-03-09T10:15:26.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073b70 con 0x7f8324005e00 2021-03-09T10:15:26.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:15:26.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.390+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:26.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.394+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:26.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.394+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:26.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.394+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:26.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.394+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:26.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.394+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380080a0 con 0x7f8338015d30 2021-03-09T10:15:26.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.394+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069320 secure :-1 s=READY pgs=7659 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334032000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:26.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.394+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340567c0 con 0x7f8338015d30 2021-03-09T10:15:26.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.394+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334036430 con 0x7f8338015d30 2021-03-09T10:15:26.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:26.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334047ad0 con 0x7f8338015d30 2021-03-09T10:15:26.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:15:26 smithi146 bash[13136]: debug 2021-03-09T10:15:26.389+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22070 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:15:31.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:15:31.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:31.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069320 secure :-1 s=READY pgs=7659 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334032000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:31.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069320 secure :-1 s=READY pgs=7659 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334032000).stop 2021-03-09T10:15:31.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338069320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:31.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069320 unknown :-1 s=CLOSED pgs=7659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:31.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:31.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:31.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:31.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073b70 con 0x7f8324003400 2021-03-09T10:15:31.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a3f0 con 0x7f8338071d10 2021-03-09T10:15:31.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021650 con 0x7f8324005e00 2021-03-09T10:15:31.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:15:31.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:31.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:15:31.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:31.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.394+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:31.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.398+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:31.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.398+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:31.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.398+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:31.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.398+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:31.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.398+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338015b60 con 0x7f8338071d10 2021-03-09T10:15:31.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.398+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7667 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834400e1c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:31.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f8338071d10 2021-03-09T10:15:31.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T10:15:31.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:31.402+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440791e0 con 0x7f8338071d10 2021-03-09T10:15:31.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:15:31 smithi146 bash[13136]: debug 2021-03-09T10:15:31.393+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22097 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:15:36.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:15:36.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:36.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7667 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834400e1c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:36.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7667 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834400e1c0).stop 2021-03-09T10:15:36.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:36.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:36.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:36.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:36.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:36.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073a10 con 0x7f8324003400 2021-03-09T10:15:36.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015a20 con 0x7f8338015d30 2021-03-09T10:15:36.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005210 con 0x7f8324005e00 2021-03-09T10:15:36.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:15:36.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:36.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:36.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:15:36.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.398+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:36.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.402+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:36.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.402+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:36.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.402+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:36.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.402+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:36.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.402+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338066330 con 0x7f8338015d30 2021-03-09T10:15:36.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.402+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069320 secure :-1 s=READY pgs=7675 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834400e1c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:36.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.402+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440783a0 con 0x7f8338015d30 2021-03-09T10:15:36.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.402+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404b780 con 0x7f8338015d30 2021-03-09T10:15:36.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:36.406+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f8338015d30 2021-03-09T10:15:36.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:15:36 smithi146 bash[13136]: debug 2021-03-09T10:15:36.397+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22122 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:15:41.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:15:41.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:41.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069320 secure :-1 s=READY pgs=7675 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834400e1c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:41.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069320 secure :-1 s=READY pgs=7675 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834400e1c0).stop 2021-03-09T10:15:41.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338069320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:41.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338069320 unknown :-1 s=CLOSED pgs=7675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:41.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:41.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:41.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:41.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d8f0 con 0x7f8324003400 2021-03-09T10:15:41.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801adc0 con 0x7f8338071d10 2021-03-09T10:15:41.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a3f0 con 0x7f8324005e00 2021-03-09T10:15:41.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:15:41.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:41.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:15:41.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:41.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.402+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:41.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.406+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:41.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.406+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:41.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.406+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:41.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.406+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:41.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.406+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338066330 con 0x7f8338071d10 2021-03-09T10:15:41.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.406+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7684 cs=0 l=1 rev1=1 rx=0x7f8334004230 tx=0x7f8334032000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:41.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.406+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404ead0 con 0x7f8338071d10 2021-03-09T10:15:41.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.406+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402db40 con 0x7f8338071d10 2021-03-09T10:15:41.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:41.410+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340191b0 con 0x7f8338071d10 2021-03-09T10:15:41.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:15:41 smithi146 bash[13136]: debug 2021-03-09T10:15:41.401+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22149 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:15:46.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:15:46.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:46.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7684 cs=0 l=1 rev1=1 rx=0x7f8334004230 tx=0x7f8334032000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:46.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7684 cs=0 l=1 rev1=1 rx=0x7f8334004230 tx=0x7f8334032000).stop 2021-03-09T10:15:46.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:46.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:46.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:46.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:46.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:46.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073a10 con 0x7f8324003400 2021-03-09T10:15:46.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007920 con 0x7f8338015d30 2021-03-09T10:15:46.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801adc0 con 0x7f8324005e00 2021-03-09T10:15:46.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:15:46.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:46.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:15:46.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:46.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.406+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338060780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:46.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.410+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:46.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.410+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:46.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.410+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:46.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.410+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:46.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.410+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e840 con 0x7f8338015d30 2021-03-09T10:15:46.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.410+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338060780 secure :-1 s=READY pgs=7693 cs=0 l=1 rev1=1 rx=0x7f8344075290 tx=0x7f834404b8f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:46.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.410+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402bdf0 con 0x7f8338015d30 2021-03-09T10:15:46.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.414+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f8338015d30 2021-03-09T10:15:46.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:46.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440791e0 con 0x7f8338015d30 2021-03-09T10:15:46.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:15:46 smithi146 bash[13136]: debug 2021-03-09T10:15:46.405+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22173 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:15:51.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338060780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:15:51.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338060780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:51.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338060780 secure :-1 s=READY pgs=7693 cs=0 l=1 rev1=1 rx=0x7f8344075290 tx=0x7f834404b8f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:51.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338060780 secure :-1 s=READY pgs=7693 cs=0 l=1 rev1=1 rx=0x7f8344075290 tx=0x7f834404b8f0).stop 2021-03-09T10:15:51.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 msgr2=0x7f8338060780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:51.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338015d30 0x7f8338060780 unknown :-1 s=CLOSED pgs=7693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338014e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064810 con 0x7f8324003400 2021-03-09T10:15:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801adc0 con 0x7f8338071d10 2021-03-09T10:15:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:15:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:15:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:15:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:51.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:51.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338014e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:51.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:51.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:51.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:51.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:51.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380107b0 con 0x7f8338071d10 2021-03-09T10:15:51.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.410+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338014e30 secure :-1 s=READY pgs=7701 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f834404b8f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:51.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.414+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344069be0 con 0x7f8338071d10 2021-03-09T10:15:51.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.414+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834401f120 con 0x7f8338071d10 2021-03-09T10:15:51.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:51.418+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aaf0 con 0x7f8338071d10 2021-03-09T10:15:51.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:15:51 smithi146 bash[13136]: debug 2021-03-09T10:15:51.409+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22201 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:15:56.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338014e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:15:56.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338014e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:56.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338014e30 secure :-1 s=READY pgs=7701 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f834404b8f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:56.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338014e30 secure :-1 s=READY pgs=7701 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f834404b8f0).stop 2021-03-09T10:15:56.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338014e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:56.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338014e30 unknown :-1 s=CLOSED pgs=7701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:56.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:56.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338069740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:56.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:56.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007920 con 0x7f8324003400 2021-03-09T10:15:56.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073a10 con 0x7f8338069320 2021-03-09T10:15:56.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801adc0 con 0x7f8324005e00 2021-03-09T10:15:56.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:15:56.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:15:56.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:56.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:56.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.413+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:56.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.417+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:56.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.417+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:56.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.417+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:56.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.417+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:56.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.417+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065a70 con 0x7f8338069320 2021-03-09T10:15:56.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.417+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338069740 secure :-1 s=READY pgs=7711 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:56.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.421+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834401f120 con 0x7f8338069320 2021-03-09T10:15:56.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.421+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834403cef0 con 0x7f8338069320 2021-03-09T10:15:56.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:15:56.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344078c40 con 0x7f8338069320 2021-03-09T10:15:56.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:15:56 smithi146 bash[13136]: debug 2021-03-09T10:15:56.413+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22225 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:16:01.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338069740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:16:01.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338069740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:01.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338069740 secure :-1 s=READY pgs=7711 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:01.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338069740 secure :-1 s=READY pgs=7711 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344021000).stop 2021-03-09T10:16:01.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338069740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:01.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338069740 unknown :-1 s=CLOSED pgs=7711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:01.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:01.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:01.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:01.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380110c0 con 0x7f8324003400 2021-03-09T10:16:01.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338071d10 2021-03-09T10:16:01.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bbe0 con 0x7f8324005e00 2021-03-09T10:16:01.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:16:01.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:16:01.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:01.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:01.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.417+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:01.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.421+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:01.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.421+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:01.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.421+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:01.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.421+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:01.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.421+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020d30 con 0x7f8338071d10 2021-03-09T10:16:01.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.421+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7719 cs=0 l=1 rev1=1 rx=0x7f8334063570 tx=0x7f8334032000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:01.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.421+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334016560 con 0x7f8338071d10 2021-03-09T10:16:01.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.421+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334016330 con 0x7f8338071d10 2021-03-09T10:16:01.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:01.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404eca0 con 0x7f8338071d10 2021-03-09T10:16:01.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:16:01 smithi146 bash[13136]: debug 2021-03-09T10:16:01.417+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22253 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:16:06.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:16:06.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:06.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7719 cs=0 l=1 rev1=1 rx=0x7f8334063570 tx=0x7f8334032000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:06.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7719 cs=0 l=1 rev1=1 rx=0x7f8334063570 tx=0x7f8334032000).stop 2021-03-09T10:16:06.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:06.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:06.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:06.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:06.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:06.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801adc0 con 0x7f8324003400 2021-03-09T10:16:06.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010b80 con 0x7f8338069320 2021-03-09T10:16:06.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:16:06.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:16:06.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:06.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:16:06.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:06.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:06.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:06.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:06.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.421+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022d80 con 0x7f8338069320 2021-03-09T10:16:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.425+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806c690 secure :-1 s=READY pgs=7727 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f8338069320 2021-03-09T10:16:06.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033030 con 0x7f8338069320 2021-03-09T10:16:06.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:06.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834401ed50 con 0x7f8338069320 2021-03-09T10:16:06.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:16:06 smithi146 bash[13136]: debug 2021-03-09T10:16:06.420+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22277 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:16:11.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:16:11.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:11.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806c690 secure :-1 s=READY pgs=7727 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:11.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806c690 secure :-1 s=READY pgs=7727 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344021000).stop 2021-03-09T10:16:11.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833806c690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:11.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806c690 unknown :-1 s=CLOSED pgs=7727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:11.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:11.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:11.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:11.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801adc0 con 0x7f8324003400 2021-03-09T10:16:11.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061200 con 0x7f8338071d10 2021-03-09T10:16:11.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380151f0 con 0x7f8324005e00 2021-03-09T10:16:11.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:16:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:16:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:11.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:11.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:11.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:11.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:11.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b140 con 0x7f8338071d10 2021-03-09T10:16:11.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.425+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7736 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334032000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:11.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833402da60 con 0x7f8338071d10 2021-03-09T10:16:11.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402da60 con 0x7f8338071d10 2021-03-09T10:16:11.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:11.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334062270 con 0x7f8338071d10 2021-03-09T10:16:11.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:16:11 smithi146 bash[13136]: debug 2021-03-09T10:16:11.424+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22304 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:16:16.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:16:16.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:16.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7736 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334032000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:16.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7736 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334032000).stop 2021-03-09T10:16:16.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:16.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:16.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:16.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:16.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:16.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:16:16.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:16.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801adc0 con 0x7f8324003400 2021-03-09T10:16:16.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061200 con 0x7f8338069320 2021-03-09T10:16:16.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:16:16.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:16:16.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:16.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:16.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:16.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:16.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:16.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:16.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.429+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e840 con 0x7f8338069320 2021-03-09T10:16:16.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801ff30 secure :-1 s=READY pgs=7745 cs=0 l=1 rev1=1 rx=0x7f833403a6d0 tx=0x7f8334032000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:16.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334003e40 con 0x7f8338069320 2021-03-09T10:16:16.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334003e40 con 0x7f8338069320 2021-03-09T10:16:16.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:16.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334062b20 con 0x7f8338069320 2021-03-09T10:16:16.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:16:16 smithi146 bash[13136]: debug 2021-03-09T10:16:16.428+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22329 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:16:21.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:16:21.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:21.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801ff30 secure :-1 s=READY pgs=7745 cs=0 l=1 rev1=1 rx=0x7f833403a6d0 tx=0x7f8334032000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:21.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801ff30 secure :-1 s=READY pgs=7745 cs=0 l=1 rev1=1 rx=0x7f833403a6d0 tx=0x7f8334032000).stop 2021-03-09T10:16:21.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801ff30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:21.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801ff30 unknown :-1 s=CLOSED pgs=7745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:21.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:21.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:21.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:21.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:16:21.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:21.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324003400 2021-03-09T10:16:21.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801adc0 con 0x7f8338071d10 2021-03-09T10:16:21.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380082f0 con 0x7f8324005e00 2021-03-09T10:16:21.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:21.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:16:21.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:21.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:21.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:21.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:21.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:21.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.433+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062960 con 0x7f8338071d10 2021-03-09T10:16:21.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.437+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7753 cs=0 l=1 rev1=1 rx=0x7f833406e040 tx=0x7f8334032000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:21.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334003bb0 con 0x7f8338071d10 2021-03-09T10:16:21.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334003e40 con 0x7f8338071d10 2021-03-09T10:16:21.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:21.441+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334062b20 con 0x7f8338071d10 2021-03-09T10:16:21.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:16:21 smithi146 bash[13136]: debug 2021-03-09T10:16:21.432+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22355 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:16:26.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:16:26.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:26.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7753 cs=0 l=1 rev1=1 rx=0x7f833406e040 tx=0x7f8334032000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:26.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7753 cs=0 l=1 rev1=1 rx=0x7f833406e040 tx=0x7f8334032000).stop 2021-03-09T10:16:26.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:26.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:26.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:26.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:26.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:16:26.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:26.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:16:26.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:26.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:26.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a180 con 0x7f8324003400 2021-03-09T10:16:26.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800cfb0 con 0x7f8338069320 2021-03-09T10:16:26.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380643b0 con 0x7f8324005e00 2021-03-09T10:16:26.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:26.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:26.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:26.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:26.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:26.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.437+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338002870 con 0x7f8338069320 2021-03-09T10:16:26.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.441+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020640 secure :-1 s=READY pgs=7763 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c039000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:26.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.449+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338069320 2021-03-09T10:16:26.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.449+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c004160 con 0x7f8338069320 2021-03-09T10:16:26.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:26.449+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0458a0 con 0x7f8338069320 2021-03-09T10:16:26.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:16:26 smithi146 bash[13136]: debug 2021-03-09T10:16:26.436+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22380 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:16:31.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338020640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:16:31.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338020640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:31.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020640 secure :-1 s=READY pgs=7763 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c039000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:31.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020640 secure :-1 s=READY pgs=7763 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c039000).stop 2021-03-09T10:16:31.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338020640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:31.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020640 unknown :-1 s=CLOSED pgs=7763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:31.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:31.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:31.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:31.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380151f0 con 0x7f8324003400 2021-03-09T10:16:31.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dc50 con 0x7f8338071d10 2021-03-09T10:16:31.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dd30 con 0x7f8324005e00 2021-03-09T10:16:31.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:16:31.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:16:31.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:31.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:31.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.441+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:31.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.445+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:31.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.445+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:31.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.445+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:31.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.445+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:31.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.445+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338013b20 con 0x7f8338071d10 2021-03-09T10:16:31.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.445+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7775 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c039000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:31.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.445+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0627c0 con 0x7f8338071d10 2021-03-09T10:16:31.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.449+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0627c0 con 0x7f8338071d10 2021-03-09T10:16:31.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:31.453+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338071d10 2021-03-09T10:16:31.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:16:31 smithi146 bash[13136]: debug 2021-03-09T10:16:31.440+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22407 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:16:36.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:16:36.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:36.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7775 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c039000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:36.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7775 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c039000).stop 2021-03-09T10:16:36.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:36.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:36.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:36.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:36.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:16:36.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:36.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:36.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d520 con 0x7f8324003400 2021-03-09T10:16:36.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014480 con 0x7f8338069320 2021-03-09T10:16:36.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c360 con 0x7f8324005e00 2021-03-09T10:16:36.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:16:36.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:36.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.445+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:36.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.449+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:36.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.449+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:36.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.449+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:36.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.449+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:36.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.449+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022d80 con 0x7f8338069320 2021-03-09T10:16:36.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.449+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 secure :-1 s=READY pgs=7781 cs=0 l=1 rev1=1 rx=0x7f8334009b40 tx=0x7f8334022000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:36.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.449+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334052450 con 0x7f8338069320 2021-03-09T10:16:36.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.453+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340191b0 con 0x7f8338069320 2021-03-09T10:16:36.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:36.453+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334058260 con 0x7f8338069320 2021-03-09T10:16:36.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:16:36 smithi146 bash[13136]: debug 2021-03-09T10:16:36.444+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22432 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:16:41.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:16:41.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:41.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 secure :-1 s=READY pgs=7781 cs=0 l=1 rev1=1 rx=0x7f8334009b40 tx=0x7f8334022000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:41.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 secure :-1 s=READY pgs=7781 cs=0 l=1 rev1=1 rx=0x7f8334009b40 tx=0x7f8334022000).stop 2021-03-09T10:16:41.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800b620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:41.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 unknown :-1 s=CLOSED pgs=7781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:41.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:41.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:41.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:16:41.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:41.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:41.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dd30 con 0x7f8324003400 2021-03-09T10:16:41.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a180 con 0x7f8338071d10 2021-03-09T10:16:41.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:16:41.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:16:41.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:41.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:41.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:41.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:41.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:41.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:41.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833805eaa0 con 0x7f8338071d10 2021-03-09T10:16:41.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.449+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7788 cs=0 l=1 rev1=1 rx=0x7f8334004f90 tx=0x7f833400bb90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:41.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.453+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334065bf0 con 0x7f8338071d10 2021-03-09T10:16:41.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.453+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401f440 con 0x7f8338071d10 2021-03-09T10:16:41.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:41.457+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334065bf0 con 0x7f8338071d10 2021-03-09T10:16:41.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:16:41 smithi146 bash[13136]: debug 2021-03-09T10:16:41.448+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22459 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:16:46.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:16:46.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:46.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7788 cs=0 l=1 rev1=1 rx=0x7f8334004f90 tx=0x7f833400bb90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:46.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7788 cs=0 l=1 rev1=1 rx=0x7f8334004f90 tx=0x7f833400bb90).stop 2021-03-09T10:16:46.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:46.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:46.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338024db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:46.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:46.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:46.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d520 con 0x7f8324003400 2021-03-09T10:16:46.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dd30 con 0x7f8338069320 2021-03-09T10:16:46.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a180 con 0x7f8324005e00 2021-03-09T10:16:46.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:16:46.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:46.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:16:46.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:46.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338024db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:46.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:46.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:46.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:46.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.452+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:46.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.456+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380739f0 con 0x7f8338069320 2021-03-09T10:16:46.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.456+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338024db0 secure :-1 s=READY pgs=7798 cs=0 l=1 rev1=1 rx=0x7f8334070f70 tx=0x7f833400bb90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:46.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.456+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334064490 con 0x7f8338069320 2021-03-09T10:16:46.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.460+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334064490 con 0x7f8338069320 2021-03-09T10:16:46.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:46.464+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334065bf0 con 0x7f8338069320 2021-03-09T10:16:46.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:16:46 smithi146 bash[13136]: debug 2021-03-09T10:16:46.452+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22484 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:16:51.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338024db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:16:51.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338024db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:51.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338024db0 secure :-1 s=READY pgs=7798 cs=0 l=1 rev1=1 rx=0x7f8334070f70 tx=0x7f833400bb90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:51.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338024db0 secure :-1 s=READY pgs=7798 cs=0 l=1 rev1=1 rx=0x7f8334070f70 tx=0x7f833400bb90).stop 2021-03-09T10:16:51.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338024db0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:51.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338024db0 unknown :-1 s=CLOSED pgs=7798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:51.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:51.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:51.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:16:51.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:51.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:51.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:16:51.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:51.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324003400 2021-03-09T10:16:51.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380151f0 con 0x7f8338071d10 2021-03-09T10:16:51.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dd30 con 0x7f8324005e00 2021-03-09T10:16:51.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:51.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:51.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:51.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:51.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:51.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.456+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338011410 con 0x7f8338071d10 2021-03-09T10:16:51.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.460+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7805 cs=0 l=1 rev1=1 rx=0x7f8334022f40 tx=0x7f833400bb90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:51.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.460+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833402da60 con 0x7f8338071d10 2021-03-09T10:16:51.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.460+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338071d10 2021-03-09T10:16:51.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:51.464+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334065bf0 con 0x7f8338071d10 2021-03-09T10:16:51.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:16:51 smithi146 bash[13136]: debug 2021-03-09T10:16:51.456+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22511 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:16:56.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:16:56.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:56.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7805 cs=0 l=1 rev1=1 rx=0x7f8334022f40 tx=0x7f833400bb90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:56.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7805 cs=0 l=1 rev1=1 rx=0x7f8334022f40 tx=0x7f833400bb90).stop 2021-03-09T10:16:56.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:56.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:56.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:56.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:56.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:16:56.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:16:56.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:56.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005410 con 0x7f8324003400 2021-03-09T10:16:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074f70 con 0x7f8338069320 2021-03-09T10:16:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380282e0 con 0x7f8324005e00 2021-03-09T10:16:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:56.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:56.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:56.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:56.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.460+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:56.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.464+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024780 con 0x7f8338069320 2021-03-09T10:16:56.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.464+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 secure :-1 s=READY pgs=7814 cs=0 l=1 rev1=1 rx=0x7f833c054d30 tx=0x7f833c039000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:56.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.464+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00f300 con 0x7f8338069320 2021-03-09T10:16:56.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.464+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00fa10 con 0x7f8338069320 2021-03-09T10:16:56.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:16:56.468+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c062dc0 con 0x7f8338069320 2021-03-09T10:16:56.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:16:56 smithi146 bash[13136]: debug 2021-03-09T10:16:56.464+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22536 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:17:01.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:17:01.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:01.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 secure :-1 s=READY pgs=7814 cs=0 l=1 rev1=1 rx=0x7f833c054d30 tx=0x7f833c039000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:01.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 secure :-1 s=READY pgs=7814 cs=0 l=1 rev1=1 rx=0x7f833c054d30 tx=0x7f833c039000).stop 2021-03-09T10:17:01.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800b620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:01.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 unknown :-1 s=CLOSED pgs=7814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:01.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:01.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:01.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:01.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:17:01.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:01.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:17:01.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:01.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:17:01.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d520 con 0x7f8338071d10 2021-03-09T10:17:01.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072540 con 0x7f8324005e00 2021-03-09T10:17:01.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:01.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:01.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:01.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:01.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:01.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.464+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024780 con 0x7f8338071d10 2021-03-09T10:17:01.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.468+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7823 cs=0 l=1 rev1=1 rx=0x7f834400e200 tx=0x7f834407af60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:01.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.468+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T10:17:01.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.468+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T10:17:01.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:01.472+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338071d10 2021-03-09T10:17:01.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:17:01 smithi146 bash[13136]: debug 2021-03-09T10:17:01.468+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22562 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:17:06.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:17:06.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:06.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7823 cs=0 l=1 rev1=1 rx=0x7f834400e200 tx=0x7f834407af60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:06.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7823 cs=0 l=1 rev1=1 rx=0x7f834400e200 tx=0x7f834407af60).stop 2021-03-09T10:17:06.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:06.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:06.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:06.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:06.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:17:06.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:06.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:06.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d520 con 0x7f8324003400 2021-03-09T10:17:06.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071b90 con 0x7f8338069320 2021-03-09T10:17:06.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:06.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015b60 con 0x7f8324005e00 2021-03-09T10:17:06.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:17:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.468+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.472+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.472+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.472+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:06.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.472+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:06.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.472+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338025640 con 0x7f8338069320 2021-03-09T10:17:06.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.472+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 secure :-1 s=READY pgs=7832 cs=0 l=1 rev1=1 rx=0x7f8334008ff0 tx=0x7f833400bb90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:06.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.472+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8338069320 2021-03-09T10:17:06.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.472+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8338069320 2021-03-09T10:17:06.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:06.476+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338069320 2021-03-09T10:17:06.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:17:06 smithi146 bash[13136]: debug 2021-03-09T10:17:06.472+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22588 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:17:11.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:17:11.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:11.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 secure :-1 s=READY pgs=7832 cs=0 l=1 rev1=1 rx=0x7f8334008ff0 tx=0x7f833400bb90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:11.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 secure :-1 s=READY pgs=7832 cs=0 l=1 rev1=1 rx=0x7f8334008ff0 tx=0x7f833400bb90).stop 2021-03-09T10:17:11.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800b620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:11.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 unknown :-1 s=CLOSED pgs=7832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:11.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:11.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:11.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:17:11.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:11.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:11.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b40 con 0x7f8324003400 2021-03-09T10:17:11.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338071d10 2021-03-09T10:17:11.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073b90 con 0x7f8324005e00 2021-03-09T10:17:11.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:17:11.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:11.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:11.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:11.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:11.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:11.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:11.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338008000 con 0x7f8338071d10 2021-03-09T10:17:11.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7840 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f833400bb90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:11.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404eeb0 con 0x7f8338071d10 2021-03-09T10:17:11.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.476+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334033eb0 con 0x7f8338071d10 2021-03-09T10:17:11.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:11.480+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833400d380 con 0x7f8338071d10 2021-03-09T10:17:11.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:17:11 smithi146 bash[13136]: debug 2021-03-09T10:17:11.475+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22614 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:17:16.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:17:16.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:16.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7840 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f833400bb90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:16.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7840 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f833400bb90).stop 2021-03-09T10:17:16.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:16.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:16.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:16.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:16.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:16.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:17:16.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:16.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:17:16.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:16.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a5a0 con 0x7f8324003400 2021-03-09T10:17:16.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013fa0 con 0x7f8338069320 2021-03-09T10:17:16.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324005e00 2021-03-09T10:17:16.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:16.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:16.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:16.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:16.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:16.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028d20 con 0x7f8338069320 2021-03-09T10:17:16.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 secure :-1 s=READY pgs=7849 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834407af60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:16.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403cd20 con 0x7f8338069320 2021-03-09T10:17:16.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.480+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033030 con 0x7f8338069320 2021-03-09T10:17:16.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:16.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834403c9b0 con 0x7f8338069320 2021-03-09T10:17:16.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:17:16 smithi146 bash[13136]: debug 2021-03-09T10:17:16.479+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22640 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:17:21.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:17:21.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:21.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 secure :-1 s=READY pgs=7849 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834407af60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 secure :-1 s=READY pgs=7849 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834407af60).stop 2021-03-09T10:17:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800b620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 unknown :-1 s=CLOSED pgs=7849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:21.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:21.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:17:21.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:21.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:21.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:17:21.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:21.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324003400 2021-03-09T10:17:21.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b40 con 0x7f8338071d10 2021-03-09T10:17:21.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064810 con 0x7f8324005e00 2021-03-09T10:17:21.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:21.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:21.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:21.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:21.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:21.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338011440 con 0x7f8338071d10 2021-03-09T10:17:21.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7857 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f833400bb90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:21.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404eeb0 con 0x7f8338071d10 2021-03-09T10:17:21.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.484+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8338071d10 2021-03-09T10:17:21.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:21.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334017130 con 0x7f8338071d10 2021-03-09T10:17:21.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:17:21 smithi146 bash[13136]: debug 2021-03-09T10:17:21.483+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22666 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:17:26.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:17:26.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:26.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7857 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f833400bb90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:26.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7857 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f833400bb90).stop 2021-03-09T10:17:26.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:26.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:26.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:26.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:26.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:17:26.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:26.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:26.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:26.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:17:26.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:26.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380282e0 con 0x7f8324003400 2021-03-09T10:17:26.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c9a0 con 0x7f8338069320 2021-03-09T10:17:26.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012870 con 0x7f8324005e00 2021-03-09T10:17:26.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:26.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:26.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:26.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:26.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801f300 con 0x7f8338069320 2021-03-09T10:17:26.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800d310 secure :-1 s=READY pgs=7866 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834407af60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:26.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.488+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:17:26.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:17:26.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:26.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:17:26.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:17:26 smithi146 bash[13136]: debug 2021-03-09T10:17:26.487+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22692 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:17:31.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:17:31.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:31.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800d310 secure :-1 s=READY pgs=7866 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834407af60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:31.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800d310 secure :-1 s=READY pgs=7866 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834407af60).stop 2021-03-09T10:17:31.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800d310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:31.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800d310 unknown :-1 s=CLOSED pgs=7866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:31.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:31.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:17:31.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:31.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:31.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:17:31.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:31.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:31.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b40 con 0x7f8324003400 2021-03-09T10:17:31.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380652b0 con 0x7f8338071d10 2021-03-09T10:17:31.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380247c0 con 0x7f8324005e00 2021-03-09T10:17:31.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:31.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:31.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:31.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:31.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:31.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b160 con 0x7f8338071d10 2021-03-09T10:17:31.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7875 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c00f970).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:31.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338071d10 2021-03-09T10:17:31.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.492+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338071d10 2021-03-09T10:17:31.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:31.496+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0040e0 con 0x7f8338071d10 2021-03-09T10:17:31.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:17:31 smithi146 bash[13136]: debug 2021-03-09T10:17:31.491+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22717 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:17:36.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:17:36.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:36.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7875 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c00f970).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:36.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7875 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c00f970).stop 2021-03-09T10:17:36.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:36.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:36.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:36.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:36.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:36.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b40 con 0x7f8324003400 2021-03-09T10:17:36.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061060 con 0x7f8338069320 2021-03-09T10:17:36.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380110c0 con 0x7f8324005e00 2021-03-09T10:17:36.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:17:36.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:36.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:17:36.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:36.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:36.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:36.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:36.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:36.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:36.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338067620 con 0x7f8338069320 2021-03-09T10:17:36.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800d310 secure :-1 s=READY pgs=7883 cs=0 l=1 rev1=1 rx=0x7f833404ebb0 tx=0x7f833403b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:36.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401f510 con 0x7f8338069320 2021-03-09T10:17:36.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334052a50 con 0x7f8338069320 2021-03-09T10:17:36.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:36.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334003d00 con 0x7f8338069320 2021-03-09T10:17:36.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:17:36 smithi146 bash[13136]: debug 2021-03-09T10:17:36.495+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22744 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:17:41.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.499+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:17:41.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.499+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:41.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.499+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800d310 secure :-1 s=READY pgs=7883 cs=0 l=1 rev1=1 rx=0x7f833404ebb0 tx=0x7f833403b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:41.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.499+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800d310 secure :-1 s=READY pgs=7883 cs=0 l=1 rev1=1 rx=0x7f833404ebb0 tx=0x7f833403b000).stop 2021-03-09T10:17:41.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800d310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:41.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800d310 unknown :-1 s=CLOSED pgs=7883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:41.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:41.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:41.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.499+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:41.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8324003400 2021-03-09T10:17:41.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073460 con 0x7f8338071d10 2021-03-09T10:17:41.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380110c0 con 0x7f8324005e00 2021-03-09T10:17:41.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.499+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:41.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.499+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:17:41.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.499+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:41.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.503+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:17:41.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.503+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:41.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.507+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:41.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.507+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:41.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.507+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:41.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.507+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:41.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.507+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338066e20 con 0x7f8338071d10 2021-03-09T10:17:41.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.507+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801cb40 secure :-1 s=READY pgs=7894 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c039000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:41.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01ac00 con 0x7f8338071d10 2021-03-09T10:17:41.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f8338071d10 2021-03-09T10:17:41.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:41.511+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f8338071d10 2021-03-09T10:17:41.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:17:41 smithi146 bash[13136]: debug 2021-03-09T10:17:41.499+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22769 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:17:46.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:17:46.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:46.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801cb40 secure :-1 s=READY pgs=7894 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c039000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:46.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801cb40 secure :-1 s=READY pgs=7894 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c039000).stop 2021-03-09T10:17:46.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801cb40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:46.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801cb40 unknown :-1 s=CLOSED pgs=7894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:46.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:46.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:46.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338018950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:46.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380282e0 con 0x7f8324003400 2021-03-09T10:17:46.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338075850 con 0x7f8338069320 2021-03-09T10:17:46.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e020 con 0x7f8324005e00 2021-03-09T10:17:46.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:17:46.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:46.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:17:46.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:46.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.503+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338018950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:46.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.507+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:46.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.507+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:46.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.507+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:46.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.507+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:46.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.507+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338074740 con 0x7f8338069320 2021-03-09T10:17:46.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.507+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338018950 secure :-1 s=READY pgs=7901 cs=0 l=1 rev1=1 rx=0x7f833c00fb10 tx=0x7f833c039000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:46.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00f770 con 0x7f8338069320 2021-03-09T10:17:46.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00f770 con 0x7f8338069320 2021-03-09T10:17:46.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:46.511+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338069320 2021-03-09T10:17:46.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:17:46 smithi146 bash[13136]: debug 2021-03-09T10:17:46.503+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22796 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:17:51.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338018950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:17:51.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338018950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338018950 secure :-1 s=READY pgs=7901 cs=0 l=1 rev1=1 rx=0x7f833c00fb10 tx=0x7f833c039000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338018950 secure :-1 s=READY pgs=7901 cs=0 l=1 rev1=1 rx=0x7f833c00fb10 tx=0x7f833c039000).stop 2021-03-09T10:17:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338018950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338018950 unknown :-1 s=CLOSED pgs=7901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014890 con 0x7f8324003400 2021-03-09T10:17:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061060 con 0x7f8338071d10 2021-03-09T10:17:51.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324005e00 2021-03-09T10:17:51.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:17:51.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:51.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:51.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:17:51.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.507+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:51.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.511+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:51.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.511+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:51.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.511+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:51.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.511+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:51.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.511+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338072540 con 0x7f8338071d10 2021-03-09T10:17:51.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.511+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7909 cs=0 l=1 rev1=1 rx=0x7f833400faf0 tx=0x7f833403b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:51.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.511+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334052520 con 0x7f8338071d10 2021-03-09T10:17:51.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.511+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340612d0 con 0x7f8338071d10 2021-03-09T10:17:51.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:51.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404eca0 con 0x7f8338071d10 2021-03-09T10:17:52.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:17:51 smithi146 bash[13136]: debug 2021-03-09T10:17:51.507+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22821 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:17:56.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:17:56.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:56.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7909 cs=0 l=1 rev1=1 rx=0x7f833400faf0 tx=0x7f833403b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:56.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7909 cs=0 l=1 rev1=1 rx=0x7f833400faf0 tx=0x7f833403b000).stop 2021-03-09T10:17:56.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:56.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:56.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:56.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:56.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:56.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338075850 con 0x7f8324003400 2021-03-09T10:17:56.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8338069320 2021-03-09T10:17:56.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014890 con 0x7f8324005e00 2021-03-09T10:17:56.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:17:56.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:56.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:17:56.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:56.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.511+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:56.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.515+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:56.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.515+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:56.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.515+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:56.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.515+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:56.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.515+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338072540 con 0x7f8338069320 2021-03-09T10:17:56.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.515+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801dee0 secure :-1 s=READY pgs=7918 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c039000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:56.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0075d0 con 0x7f8338069320 2021-03-09T10:17:56.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0075d0 con 0x7f8338069320 2021-03-09T10:17:56.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:17:56.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338069320 2021-03-09T10:17:57.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:17:56 smithi146 bash[13136]: debug 2021-03-09T10:17:56.511+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22848 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:18:01.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.515+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:18:01.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.515+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:01.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.515+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801dee0 secure :-1 s=READY pgs=7918 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c039000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:01.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.515+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801dee0 secure :-1 s=READY pgs=7918 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c039000).stop 2021-03-09T10:18:01.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801dee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:01.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.515+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801dee0 unknown :-1 s=CLOSED pgs=7918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.515+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.515+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.515+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005fa0 con 0x7f8324003400 2021-03-09T10:18:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013ac0 con 0x7f8338071d10 2021-03-09T10:18:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.515+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014890 con 0x7f8324005e00 2021-03-09T10:18:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.515+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:18:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.515+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.515+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:18:01.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.515+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:01.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.519+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:01.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.519+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:01.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.519+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:01.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.519+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:01.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.519+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:01.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.519+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338061010 con 0x7f8338071d10 2021-03-09T10:18:01.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.523+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7930 cs=0 l=1 rev1=1 rx=0x7f833403c050 tx=0x7f833403b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:01.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334052450 con 0x7f8338071d10 2021-03-09T10:18:01.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334033e40 con 0x7f8338071d10 2021-03-09T10:18:01.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:01.527+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404eca0 con 0x7f8338071d10 2021-03-09T10:18:02.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:18:01 smithi146 bash[13136]: debug 2021-03-09T10:18:01.515+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22873 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:18:06.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.519+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:18:06.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.519+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:06.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.519+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7930 cs=0 l=1 rev1=1 rx=0x7f833403c050 tx=0x7f833403b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.519+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7930 cs=0 l=1 rev1=1 rx=0x7f833403c050 tx=0x7f833403b000).stop 2021-03-09T10:18:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.519+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.519+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.519+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.519+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:06.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.519+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:18:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.519+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.519+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:18:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.519+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.519+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338072540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.519+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023340 con 0x7f8324003400 2021-03-09T10:18:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.519+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014890 con 0x7f8338069320 2021-03-09T10:18:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.519+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b40 con 0x7f8324005e00 2021-03-09T10:18:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.523+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.523+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:06.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.523+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:06.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.523+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:06.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.523+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:06.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.523+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022d80 con 0x7f8338069320 2021-03-09T10:18:06.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.523+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338072540 secure :-1 s=READY pgs=7935 cs=0 l=1 rev1=1 rx=0x7f833c035520 tx=0x7f833c039000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:06.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c09a310 con 0x7f8338069320 2021-03-09T10:18:06.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c007560 con 0x7f8338069320 2021-03-09T10:18:06.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:06.527+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338069320 2021-03-09T10:18:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:18:06 smithi146 bash[13136]: debug 2021-03-09T10:18:06.519+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22900 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:18:11.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338072540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:18:11.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338072540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:11.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338072540 secure :-1 s=READY pgs=7935 cs=0 l=1 rev1=1 rx=0x7f833c035520 tx=0x7f833c039000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:11.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338072540 secure :-1 s=READY pgs=7935 cs=0 l=1 rev1=1 rx=0x7f833c035520 tx=0x7f833c039000).stop 2021-03-09T10:18:11.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338072540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:11.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338072540 unknown :-1 s=CLOSED pgs=7935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:11.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:11.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:11.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:11.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065a50 con 0x7f8324003400 2021-03-09T10:18:11.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014890 con 0x7f8338071d10 2021-03-09T10:18:11.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b40 con 0x7f8324005e00 2021-03-09T10:18:11.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:18:11.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:11.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:18:11.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:11.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.523+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:11.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.527+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:11.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.527+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:11.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.527+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:11.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.527+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:11.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.527+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8338071d10 2021-03-09T10:18:11.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.527+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7944 cs=0 l=1 rev1=1 rx=0x7f833401ade0 tx=0x7f833403b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:11.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.527+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833402b130 con 0x7f8338071d10 2021-03-09T10:18:11.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334003530 con 0x7f8338071d10 2021-03-09T10:18:11.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:11.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338071d10 2021-03-09T10:18:12.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:18:11 smithi146 bash[13136]: debug 2021-03-09T10:18:11.527+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22925 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:18:16.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.527+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:18:16.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.527+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:16.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.527+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7944 cs=0 l=1 rev1=1 rx=0x7f833401ade0 tx=0x7f833403b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:16.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.527+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7944 cs=0 l=1 rev1=1 rx=0x7f833401ade0 tx=0x7f833403b000).stop 2021-03-09T10:18:16.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.527+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:16.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.527+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:16.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.527+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:16.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.527+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:16.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.527+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:18:16.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.531+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:16.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.531+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:16.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:18:16.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338075850 con 0x7f8338069320 2021-03-09T10:18:16.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b3c0 con 0x7f8324005e00 2021-03-09T10:18:16.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.531+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:16.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.531+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:18:16.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.531+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:16.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.531+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:16.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.531+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:16.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.531+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:16.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.531+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:16.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.531+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062960 con 0x7f8338069320 2021-03-09T10:18:16.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.531+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce90 secure :-1 s=READY pgs=7953 cs=0 l=1 rev1=1 rx=0x7f83340473f0 tx=0x7f833403b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:16.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833406d9a0 con 0x7f8338069320 2021-03-09T10:18:16.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340191b0 con 0x7f8338069320 2021-03-09T10:18:16.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:16.535+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334071cf0 con 0x7f8338069320 2021-03-09T10:18:17.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:18:16 smithi146 bash[13136]: debug 2021-03-09T10:18:16.530+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22952 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:18:21.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833806ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:18:21.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833806ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:21.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce90 secure :-1 s=READY pgs=7953 cs=0 l=1 rev1=1 rx=0x7f83340473f0 tx=0x7f833403b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:21.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce90 secure :-1 s=READY pgs=7953 cs=0 l=1 rev1=1 rx=0x7f83340473f0 tx=0x7f833403b000).stop 2021-03-09T10:18:21.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833806ce90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:21.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce90 unknown :-1 s=CLOSED pgs=7953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:21.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:21.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:21.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:21.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014890 con 0x7f8324003400 2021-03-09T10:18:21.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:18:21.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b40 con 0x7f8338071d10 2021-03-09T10:18:21.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338075850 con 0x7f8324005e00 2021-03-09T10:18:21.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:21.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:18:21.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:21.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.531+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:21.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.535+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:21.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.535+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:21.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.535+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:21.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.535+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:21.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.535+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338071d10 2021-03-09T10:18:21.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.535+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7961 cs=0 l=1 rev1=1 rx=0x7f8338017450 tx=0x7f834407af60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:21.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.539+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T10:18:21.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.539+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T10:18:21.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:21.539+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aaf0 con 0x7f8338071d10 2021-03-09T10:18:22.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:18:21 smithi146 bash[13136]: debug 2021-03-09T10:18:21.534+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 22977 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:18:26.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.534+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:18:26.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.534+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:26.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.534+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7961 cs=0 l=1 rev1=1 rx=0x7f8338017450 tx=0x7f834407af60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:26.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.534+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7961 cs=0 l=1 rev1=1 rx=0x7f8338017450 tx=0x7f834407af60).stop 2021-03-09T10:18:26.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.534+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:26.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.534+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:26.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.534+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:26.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.534+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380720f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:26.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.534+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:26.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.534+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806f030 con 0x7f8324003400 2021-03-09T10:18:26.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.534+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065a50 con 0x7f8338069320 2021-03-09T10:18:26.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.534+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014890 con 0x7f8324005e00 2021-03-09T10:18:26.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.534+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:18:26.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.538+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:26.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.538+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:18:26.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.538+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:26.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.538+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:26.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.538+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:26.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.538+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:26.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.538+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:26.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.538+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:26.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.538+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8338069320 2021-03-09T10:18:26.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.538+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380720f0 secure :-1 s=READY pgs=7970 cs=0 l=1 rev1=1 rx=0x7f834407afd0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:26.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.542+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344010a80 con 0x7f8338069320 2021-03-09T10:18:26.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.542+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440106d0 con 0x7f8338069320 2021-03-09T10:18:26.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:26.546+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440179e0 con 0x7f8338069320 2021-03-09T10:18:27.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:18:26 smithi146 bash[13136]: debug 2021-03-09T10:18:26.538+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23004 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:18:31.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:18:31.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:31.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380720f0 secure :-1 s=READY pgs=7970 cs=0 l=1 rev1=1 rx=0x7f834407afd0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:31.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380720f0 secure :-1 s=READY pgs=7970 cs=0 l=1 rev1=1 rx=0x7f834407afd0 tx=0x7f834404c330).stop 2021-03-09T10:18:31.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380720f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:31.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380720f0 unknown :-1 s=CLOSED pgs=7970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:31.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:31.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:31.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:31.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026b40 con 0x7f8324003400 2021-03-09T10:18:31.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338071d10 2021-03-09T10:18:31.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014890 con 0x7f8324005e00 2021-03-09T10:18:31.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:18:31.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:31.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:18:31.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:31.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:31.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:31.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:31.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:31.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:31.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338016650 con 0x7f8338071d10 2021-03-09T10:18:31.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.542+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7979 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c039000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:31.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.546+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338071d10 2021-03-09T10:18:31.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.546+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f8338071d10 2021-03-09T10:18:31.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:31.546+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00f4c0 con 0x7f8338071d10 2021-03-09T10:18:32.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:18:31 smithi146 bash[13136]: debug 2021-03-09T10:18:31.542+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23029 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:18:36.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.542+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:18:36.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.542+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:36.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.542+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7979 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c039000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:36.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.542+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7979 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c039000).stop 2021-03-09T10:18:36.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.542+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:36.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.542+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:36.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.542+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:36.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.546+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:18:36.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.546+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:36.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.546+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:36.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.546+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:36.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.546+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014890 con 0x7f8324003400 2021-03-09T10:18:36.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.546+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028a70 con 0x7f8338069320 2021-03-09T10:18:36.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.546+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f500 con 0x7f8324005e00 2021-03-09T10:18:36.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.546+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:18:36.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.546+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:36.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.546+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:36.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.546+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:36.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.546+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:36.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.546+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:36.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.546+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:36.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.546+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338069320 2021-03-09T10:18:36.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.546+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 secure :-1 s=READY pgs=7987 cs=0 l=1 rev1=1 rx=0x7f833c0075a0 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:36.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.550+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c007610 con 0x7f8338069320 2021-03-09T10:18:36.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.550+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0553a0 con 0x7f8338069320 2021-03-09T10:18:36.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:36.554+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c061480 con 0x7f8338069320 2021-03-09T10:18:37.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:18:36 smithi146 bash[13136]: debug 2021-03-09T10:18:36.546+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23056 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:18:41.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:18:41.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:41.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 secure :-1 s=READY pgs=7987 cs=0 l=1 rev1=1 rx=0x7f833c0075a0 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:41.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 secure :-1 s=READY pgs=7987 cs=0 l=1 rev1=1 rx=0x7f833c0075a0 tx=0x7f833c00c2c0).stop 2021-03-09T10:18:41.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:41.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 unknown :-1 s=CLOSED pgs=7987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:41.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:41.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:41.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:41.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014890 con 0x7f8324003400 2021-03-09T10:18:41.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069fa0 con 0x7f8338071d10 2021-03-09T10:18:41.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380282e0 con 0x7f8324005e00 2021-03-09T10:18:41.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:18:41.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:18:41.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:41.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:41.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:41.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:41.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:41.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:41.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:41.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007730 con 0x7f8338071d10 2021-03-09T10:18:41.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.550+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7996 cs=0 l=1 rev1=1 rx=0x7f83440134f0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:41.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.554+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344017630 con 0x7f8338071d10 2021-03-09T10:18:41.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.554+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T10:18:41.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:41.554+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338071d10 2021-03-09T10:18:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:18:41 smithi146 bash[13136]: debug 2021-03-09T10:18:41.550+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23081 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:18:46.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.550+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:18:46.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.550+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:46.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.550+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7996 cs=0 l=1 rev1=1 rx=0x7f83440134f0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:46.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.550+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=7996 cs=0 l=1 rev1=1 rx=0x7f83440134f0 tx=0x7f834404c330).stop 2021-03-09T10:18:46.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:46.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=7996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:46.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380264d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:46.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:46.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:46.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:18:46.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8338069320 2021-03-09T10:18:46.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801fd80 con 0x7f8324005e00 2021-03-09T10:18:46.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:18:46.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:18:46.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:46.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:46.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380264d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:46.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:46.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:46.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:46.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:46.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380107b0 con 0x7f8338069320 2021-03-09T10:18:46.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.554+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380264d0 secure :-1 s=READY pgs=8005 cs=0 l=1 rev1=1 rx=0x7f833404f420 tx=0x7f833403b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:46.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.558+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334064bb0 con 0x7f8338069320 2021-03-09T10:18:46.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.558+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334062480 con 0x7f8338069320 2021-03-09T10:18:46.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:46.558+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334056d90 con 0x7f8338069320 2021-03-09T10:18:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:18:46 smithi146 bash[13136]: debug 2021-03-09T10:18:46.554+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23108 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:18:51.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380264d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:18:51.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380264d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380264d0 secure :-1 s=READY pgs=8005 cs=0 l=1 rev1=1 rx=0x7f833404f420 tx=0x7f833403b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380264d0 secure :-1 s=READY pgs=8005 cs=0 l=1 rev1=1 rx=0x7f833404f420 tx=0x7f833403b000).stop 2021-03-09T10:18:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380264d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380264d0 unknown :-1 s=CLOSED pgs=8005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014890 con 0x7f8324003400 2021-03-09T10:18:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060fe0 con 0x7f8338071d10 2021-03-09T10:18:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8324005e00 2021-03-09T10:18:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:18:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:51.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:18:51.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017c60 con 0x7f8338071d10 2021-03-09T10:18:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.558+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8013 cs=0 l=1 rev1=1 rx=0x7f8334046790 tx=0x7f833403b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404eca0 con 0x7f8338071d10 2021-03-09T10:18:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401f440 con 0x7f8338071d10 2021-03-09T10:18:51.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:51.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334062350 con 0x7f8338071d10 2021-03-09T10:18:52.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:18:51 smithi146 bash[13136]: debug 2021-03-09T10:18:51.558+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23133 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:18:56.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:18:56.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:56.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8013 cs=0 l=1 rev1=1 rx=0x7f8334046790 tx=0x7f833403b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:56.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8013 cs=0 l=1 rev1=1 rx=0x7f8334046790 tx=0x7f833403b000).stop 2021-03-09T10:18:56.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:56.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:56.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:56.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338021660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:56.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:56.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380684c0 con 0x7f8324003400 2021-03-09T10:18:56.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338069320 2021-03-09T10:18:56.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806f030 con 0x7f8324005e00 2021-03-09T10:18:56.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:18:56.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:56.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:18:56.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:56.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.562+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338021660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:56.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.566+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:56.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.566+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:56.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.566+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:56.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.566+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:56.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.566+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338069320 2021-03-09T10:18:56.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.566+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338021660 secure :-1 s=READY pgs=8027 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:56.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f8338069320 2021-03-09T10:18:56.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f8338069320 2021-03-09T10:18:56.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:18:56.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:18:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:18:56 smithi146 bash[13136]: debug 2021-03-09T10:18:56.562+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23159 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:19:01.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338021660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:19:01.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338021660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:01.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338021660 secure :-1 s=READY pgs=8027 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:01.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338021660 secure :-1 s=READY pgs=8027 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f834404c330).stop 2021-03-09T10:19:01.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338021660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:01.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338021660 unknown :-1 s=CLOSED pgs=8027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:01.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:01.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:01.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:01.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073040 con 0x7f8324003400 2021-03-09T10:19:01.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005570 con 0x7f8338071d10 2021-03-09T10:19:01.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324005e00 2021-03-09T10:19:01.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:19:01.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:01.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:19:01.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:01.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:01.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:01.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:01.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:01.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:01.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064330 con 0x7f8338071d10 2021-03-09T10:19:01.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.566+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8031 cs=0 l=1 rev1=1 rx=0x7f833c09b040 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:01.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0553a0 con 0x7f8338071d10 2021-03-09T10:19:01.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0070d0 con 0x7f8338071d10 2021-03-09T10:19:01.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:01.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338071d10 2021-03-09T10:19:02.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:19:01 smithi146 bash[13136]: debug 2021-03-09T10:19:01.566+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23185 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:19:06.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:19:06.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:06.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8031 cs=0 l=1 rev1=1 rx=0x7f833c09b040 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:06.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8031 cs=0 l=1 rev1=1 rx=0x7f833c09b040 tx=0x7f833c00c2c0).stop 2021-03-09T10:19:06.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:06.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:06.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:06.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338014d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:06.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:06.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:19:06.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017340 con 0x7f8338069320 2021-03-09T10:19:06.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a90 con 0x7f8324005e00 2021-03-09T10:19:06.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:19:06.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:06.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:19:06.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:06.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338014d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:06.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:06.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:06.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:06.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:06.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.570+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014840 con 0x7f8338069320 2021-03-09T10:19:06.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.574+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338014d50 secure :-1 s=READY pgs=8039 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:06.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:19:06.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338069320 2021-03-09T10:19:06.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:06.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:19:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:19:06 smithi146 bash[13136]: debug 2021-03-09T10:19:06.570+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23211 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:19:11.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338014d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:19:11.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338014d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:11.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338014d50 secure :-1 s=READY pgs=8039 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:11.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338014d50 secure :-1 s=READY pgs=8039 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f834404c330).stop 2021-03-09T10:19:11.576 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338014d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:11.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338014d50 unknown :-1 s=CLOSED pgs=8039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:11.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:11.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:11.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:11.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806f030 con 0x7f8324003400 2021-03-09T10:19:11.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013ac0 con 0x7f8338071d10 2021-03-09T10:19:11.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338075040 con 0x7f8324005e00 2021-03-09T10:19:11.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:19:11.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:11.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:19:11.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:11.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:11.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:11.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:11.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:11.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:11.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e530 con 0x7f8338071d10 2021-03-09T10:19:11.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.574+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8048 cs=0 l=1 rev1=1 rx=0x7f833c0548a0 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:11.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f8338071d10 2021-03-09T10:19:11.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.578+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0553a0 con 0x7f8338071d10 2021-03-09T10:19:11.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:11.582+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338071d10 2021-03-09T10:19:12.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:19:11 smithi146 bash[13136]: debug 2021-03-09T10:19:11.574+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23237 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:19:16.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:19:16.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:16.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8048 cs=0 l=1 rev1=1 rx=0x7f833c0548a0 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:16.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8048 cs=0 l=1 rev1=1 rx=0x7f833c0548a0 tx=0x7f833c00c2c0).stop 2021-03-09T10:19:16.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:16.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:16.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:16.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338015570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:16.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:16.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:19:16.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c300 con 0x7f8338069320 2021-03-09T10:19:16.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801fd80 con 0x7f8324005e00 2021-03-09T10:19:16.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:19:16.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:19:16.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:16.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:16.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338015570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:16.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:16.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:16.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:16.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:16.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.577+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338023340 con 0x7f8338069320 2021-03-09T10:19:16.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338015570 secure :-1 s=READY pgs=8057 cs=0 l=1 rev1=1 rx=0x7f8344078ee0 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:16.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:19:16.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019e60 con 0x7f8338069320 2021-03-09T10:19:16.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:16.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:19:17.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:19:16 smithi146 bash[13136]: debug 2021-03-09T10:19:16.578+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23259 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:19:21.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338015570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:19:21.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338015570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:21.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338015570 secure :-1 s=READY pgs=8057 cs=0 l=1 rev1=1 rx=0x7f8344078ee0 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:21.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338015570 secure :-1 s=READY pgs=8057 cs=0 l=1 rev1=1 rx=0x7f8344078ee0 tx=0x7f834404c330).stop 2021-03-09T10:19:21.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338015570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:21.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338015570 unknown :-1 s=CLOSED pgs=8057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:21.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:21.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:21.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:21.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801fd80 con 0x7f8324003400 2021-03-09T10:19:21.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a6a0 con 0x7f8338071d10 2021-03-09T10:19:21.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071bd0 con 0x7f8324005e00 2021-03-09T10:19:21.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:19:21.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:21.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:19:21.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:21.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:21.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:21.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:21.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:21.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:21.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338003470 con 0x7f8338071d10 2021-03-09T10:19:21.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8065 cs=0 l=1 rev1=1 rx=0x7f834403ab70 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:21.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T10:19:21.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T10:19:21.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:21.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405caf0 con 0x7f8338071d10 2021-03-09T10:19:22.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:19:21 smithi146 bash[13136]: debug 2021-03-09T10:19:21.582+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23289 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:19:26.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:19:26.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:26.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8065 cs=0 l=1 rev1=1 rx=0x7f834403ab70 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:26.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8065 cs=0 l=1 rev1=1 rx=0x7f834403ab70 tx=0x7f834404c330).stop 2021-03-09T10:19:26.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:26.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:26.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338010820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:26.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:26.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:26.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:19:26.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013ac0 con 0x7f8338069320 2021-03-09T10:19:26.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a6a0 con 0x7f8324005e00 2021-03-09T10:19:26.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:19:26.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:26.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338010820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:26.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:19:26.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:26.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.589+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:26.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.589+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:26.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.589+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:26.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.589+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:26.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.589+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338069320 2021-03-09T10:19:26.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.589+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338010820 secure :-1 s=READY pgs=8074 cs=0 l=1 rev1=1 rx=0x7f8334047e70 tx=0x7f833403b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:26.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334031130 con 0x7f8338069320 2021-03-09T10:19:26.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401f440 con 0x7f8338069320 2021-03-09T10:19:26.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:26.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404b380 con 0x7f8338069320 2021-03-09T10:19:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:19:26 smithi146 bash[13136]: debug 2021-03-09T10:19:26.585+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23314 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:19:31.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338010820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:19:31.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338010820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:31.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338010820 secure :-1 s=READY pgs=8074 cs=0 l=1 rev1=1 rx=0x7f8334047e70 tx=0x7f833403b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:31.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338010820 secure :-1 s=READY pgs=8074 cs=0 l=1 rev1=1 rx=0x7f8334047e70 tx=0x7f833403b000).stop 2021-03-09T10:19:31.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338010820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:31.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338010820 unknown :-1 s=CLOSED pgs=8074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:31.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:31.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:31.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:31.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069fa0 con 0x7f8324003400 2021-03-09T10:19:31.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c290 con 0x7f8338071d10 2021-03-09T10:19:31.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806c890 con 0x7f8324005e00 2021-03-09T10:19:31.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:19:31.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:31.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:19:31.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:31.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:31.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:31.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:31.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:31.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:31.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f8338071d10 2021-03-09T10:19:31.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8083 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:31.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04a3d0 con 0x7f8338071d10 2021-03-09T10:19:31.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c007210 con 0x7f8338071d10 2021-03-09T10:19:31.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:31.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00f300 con 0x7f8338071d10 2021-03-09T10:19:32.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:19:31 smithi146 bash[13136]: debug 2021-03-09T10:19:31.593+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23341 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:19:36.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:19:36.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:36.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8083 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:36.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8083 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c00c2c0).stop 2021-03-09T10:19:36.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:36.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:36.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:36.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338072e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:36.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.593+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:36.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006320 con 0x7f8324003400 2021-03-09T10:19:36.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338069320 2021-03-09T10:19:36.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f1f0 con 0x7f8324005e00 2021-03-09T10:19:36.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:19:36.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.597+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:19:36.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338072e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:36.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.597+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:36.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.597+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:36.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.597+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:36.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:36.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.597+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:36.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:36.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.597+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338011eb0 con 0x7f8338069320 2021-03-09T10:19:36.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338072e00 secure :-1 s=READY pgs=8091 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:36.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:19:36.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338069320 2021-03-09T10:19:36.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:36.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834403ada0 con 0x7f8338069320 2021-03-09T10:19:37.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:19:36 smithi146 bash[13136]: debug 2021-03-09T10:19:36.597+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23366 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:19:41.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.601+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338072e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:19:41.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.601+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338072e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:41.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.601+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338072e00 secure :-1 s=READY pgs=8091 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:41.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.601+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338072e00 secure :-1 s=READY pgs=8091 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404c330).stop 2021-03-09T10:19:41.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338072e00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:41.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.601+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338072e00 unknown :-1 s=CLOSED pgs=8091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:41.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.601+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:41.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.601+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:41.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.601+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:41.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.601+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:41.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a0d0 con 0x7f8324003400 2021-03-09T10:19:41.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f1f0 con 0x7f8338071d10 2021-03-09T10:19:41.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071b90 con 0x7f8324005e00 2021-03-09T10:19:41.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.605+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:41.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.605+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:19:41.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.605+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:41.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.605+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:41.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.605+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:41.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.605+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:41.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.605+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802bb50 con 0x7f8338071d10 2021-03-09T10:19:41.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.605+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8102 cs=0 l=1 rev1=1 rx=0x7f833406d520 tx=0x7f833403b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:41.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338071d10 2021-03-09T10:19:41.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833406d180 con 0x7f8338071d10 2021-03-09T10:19:41.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:41.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404b380 con 0x7f8338071d10 2021-03-09T10:19:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:19:41 smithi146 bash[13136]: debug 2021-03-09T10:19:41.601+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23393 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:19:46.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:19:46.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:46.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8102 cs=0 l=1 rev1=1 rx=0x7f833406d520 tx=0x7f833403b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:46.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8102 cs=0 l=1 rev1=1 rx=0x7f833406d520 tx=0x7f833403b000).stop 2021-03-09T10:19:46.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:46.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:46.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:46.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:46.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:46.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:19:46.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016b30 con 0x7f8338069320 2021-03-09T10:19:46.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006320 con 0x7f8324005e00 2021-03-09T10:19:46.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:19:46.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:46.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:19:46.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:46.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:46.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:46.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:46.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:46.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:46.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338069320 2021-03-09T10:19:46.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.605+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a7c0 secure :-1 s=READY pgs=8110 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:46.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400f960 con 0x7f8338069320 2021-03-09T10:19:46.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338069320 2021-03-09T10:19:46.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:46.613+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:19:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:19:46 smithi146 bash[13136]: debug 2021-03-09T10:19:46.605+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23418 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:19:51.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:19:51.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:51.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a7c0 secure :-1 s=READY pgs=8110 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:51.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a7c0 secure :-1 s=READY pgs=8110 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404c330).stop 2021-03-09T10:19:51.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800a7c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:51.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a7c0 unknown :-1 s=CLOSED pgs=8110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:51.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:51.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338073440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:51.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:51.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a0d0 con 0x7f8324003400 2021-03-09T10:19:51.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016d70 con 0x7f8338071d10 2021-03-09T10:19:51.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e870 con 0x7f8324005e00 2021-03-09T10:19:51.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:19:51.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:51.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:19:51.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:51.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338073440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:51.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:51.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:51.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:51.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:51.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338025240 con 0x7f8338071d10 2021-03-09T10:19:51.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.609+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338073440 secure :-1 s=READY pgs=8117 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:51.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.613+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004560 con 0x7f8338071d10 2021-03-09T10:19:51.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.613+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344078c30 con 0x7f8338071d10 2021-03-09T10:19:51.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:51.613+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400fb70 con 0x7f8338071d10 2021-03-09T10:19:52.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:19:51 smithi146 bash[13136]: debug 2021-03-09T10:19:51.609+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23445 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:19:56.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.609+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:19:56.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.609+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:56.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.609+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338073440 secure :-1 s=READY pgs=8117 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:56.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.609+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338073440 secure :-1 s=READY pgs=8117 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404c330).stop 2021-03-09T10:19:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338073440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.609+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338073440 unknown :-1 s=CLOSED pgs=8117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.609+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.609+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.609+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806c8d0 con 0x7f8324003400 2021-03-09T10:19:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c300 con 0x7f8338069320 2021-03-09T10:19:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006320 con 0x7f8324005e00 2021-03-09T10:19:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.609+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:19:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.609+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:19:56.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.609+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:56.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.609+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:56.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.613+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.613+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.613+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.613+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.613+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:56.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.613+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022d40 con 0x7f8338069320 2021-03-09T10:19:56.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.613+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 secure :-1 s=READY pgs=8126 cs=0 l=1 rev1=1 rx=0x7f83340312a0 tx=0x7f833403b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:56.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.617+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338069320 2021-03-09T10:19:56.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.617+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338069320 2021-03-09T10:19:56.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:19:56.621+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833406d180 con 0x7f8338069320 2021-03-09T10:19:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:19:56 smithi146 bash[13136]: debug 2021-03-09T10:19:56.613+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23470 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:20:01.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:20:01.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:01.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 secure :-1 s=READY pgs=8126 cs=0 l=1 rev1=1 rx=0x7f83340312a0 tx=0x7f833403b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:01.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 secure :-1 s=READY pgs=8126 cs=0 l=1 rev1=1 rx=0x7f83340312a0 tx=0x7f833403b000).stop 2021-03-09T10:20:01.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802cd20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:01.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 unknown :-1 s=CLOSED pgs=8126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:01.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:01.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:01.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:01.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a0d0 con 0x7f8324003400 2021-03-09T10:20:01.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c9a0 con 0x7f8338071d10 2021-03-09T10:20:01.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806c8d0 con 0x7f8324005e00 2021-03-09T10:20:01.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:20:01.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:20:01.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:01.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:01.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:01.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:01.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:01.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:01.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:01.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.617+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380060f0 con 0x7f8338071d10 2021-03-09T10:20:01.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.621+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8135 cs=0 l=1 rev1=1 rx=0x7f833c0544d0 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:01.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.621+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f8338071d10 2021-03-09T10:20:01.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.621+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338071d10 2021-03-09T10:20:01.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:01.625+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c022730 con 0x7f8338071d10 2021-03-09T10:20:02.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:20:01 smithi146 bash[13136]: debug 2021-03-09T10:20:01.617+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23497 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:20:06.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:20:06.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:06.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8135 cs=0 l=1 rev1=1 rx=0x7f833c0544d0 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:06.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8135 cs=0 l=1 rev1=1 rx=0x7f833c0544d0 tx=0x7f833c00c2c0).stop 2021-03-09T10:20:06.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:06.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:06.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:06.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:06.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:06.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c300 con 0x7f8324003400 2021-03-09T10:20:06.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016d70 con 0x7f8338069320 2021-03-09T10:20:06.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a0d0 con 0x7f8324005e00 2021-03-09T10:20:06.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:20:06.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:20:06.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:06.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:06.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:06.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:06.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:06.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:06.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:06.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338069320 2021-03-09T10:20:06.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.620+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 secure :-1 s=READY pgs=8143 cs=0 l=1 rev1=1 rx=0x7f833401cce0 tx=0x7f833403b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:06.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401cb20 con 0x7f8338069320 2021-03-09T10:20:06.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401cb20 con 0x7f8338069320 2021-03-09T10:20:06.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:06.628+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334056170 con 0x7f8338069320 2021-03-09T10:20:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:20:06 smithi146 bash[13136]: debug 2021-03-09T10:20:06.621+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23522 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:20:11.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:20:11.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:11.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 secure :-1 s=READY pgs=8143 cs=0 l=1 rev1=1 rx=0x7f833401cce0 tx=0x7f833403b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:11.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 secure :-1 s=READY pgs=8143 cs=0 l=1 rev1=1 rx=0x7f833401cce0 tx=0x7f833403b000).stop 2021-03-09T10:20:11.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802cd20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:11.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 unknown :-1 s=CLOSED pgs=8143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:11.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:11.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:11.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:11.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007ea0 con 0x7f8324003400 2021-03-09T10:20:11.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8338071d10 2021-03-09T10:20:11.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e890 con 0x7f8324005e00 2021-03-09T10:20:11.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:20:11.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:20:11.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:11.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:11.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:11.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:11.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:11.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:11.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:11.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.624+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338025240 con 0x7f8338071d10 2021-03-09T10:20:11.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.628+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8152 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:11.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.628+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c09a700 con 0x7f8338071d10 2021-03-09T10:20:11.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c09a700 con 0x7f8338071d10 2021-03-09T10:20:11.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:11.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338071d10 2021-03-09T10:20:12.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:20:11 smithi146 bash[13136]: debug 2021-03-09T10:20:11.625+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23549 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:20:16.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:20:16.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:16.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8152 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:16.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8152 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c00c2c0).stop 2021-03-09T10:20:16.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:16.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:16.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:16.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:20:16.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:16.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:16.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:16.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c290 con 0x7f8324003400 2021-03-09T10:20:16.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a0d0 con 0x7f8338069320 2021-03-09T10:20:16.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007cf0 con 0x7f8324005e00 2021-03-09T10:20:16.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:20:16.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:16.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:16.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:16.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:16.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:16.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:16.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278f0 con 0x7f8338069320 2021-03-09T10:20:16.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.632+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 secure :-1 s=READY pgs=8163 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:16.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403a5a0 con 0x7f8338069320 2021-03-09T10:20:16.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338069320 2021-03-09T10:20:16.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:16.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400fb70 con 0x7f8338069320 2021-03-09T10:20:17.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:20:16 smithi146 bash[13136]: debug 2021-03-09T10:20:16.633+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23574 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:20:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:20:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 secure :-1 s=READY pgs=8163 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 secure :-1 s=READY pgs=8163 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f834404c330).stop 2021-03-09T10:20:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802cd20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802cd20 unknown :-1 s=CLOSED pgs=8163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:20:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8338071d10 2021-03-09T10:20:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806ca80 con 0x7f8324005e00 2021-03-09T10:20:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:20:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:20:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.632+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:21.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:21.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:21.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:21.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:21.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022d40 con 0x7f8338071d10 2021-03-09T10:20:21.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8169 cs=0 l=1 rev1=1 rx=0x7f8338031260 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:21.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01c710 con 0x7f8338071d10 2021-03-09T10:20:21.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.640+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00c330 con 0x7f8338071d10 2021-03-09T10:20:21.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:21.640+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c330 con 0x7f8338071d10 2021-03-09T10:20:22.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:20:21 smithi146 bash[13136]: debug 2021-03-09T10:20:21.637+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23601 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:20:26.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:20:26.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:26.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8169 cs=0 l=1 rev1=1 rx=0x7f8338031260 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:26.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8169 cs=0 l=1 rev1=1 rx=0x7f8338031260 tx=0x7f833c00c2c0).stop 2021-03-09T10:20:26.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:26.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:26.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338004e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:26.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:26.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:26.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c300 con 0x7f8324003400 2021-03-09T10:20:26.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c290 con 0x7f8338069320 2021-03-09T10:20:26.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b8c0 con 0x7f8324005e00 2021-03-09T10:20:26.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:20:26.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:26.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:20:26.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:26.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338004e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:26.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:26.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:26.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:26.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:26.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338011440 con 0x7f8338069320 2021-03-09T10:20:26.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.640+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338004e30 secure :-1 s=READY pgs=8178 cs=0 l=1 rev1=1 rx=0x7f833404e040 tx=0x7f833404b2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:26.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.644+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338069320 2021-03-09T10:20:26.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.644+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338069320 2021-03-09T10:20:26.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:26.644+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403a1f0 con 0x7f8338069320 2021-03-09T10:20:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:20:26 smithi146 bash[13136]: debug 2021-03-09T10:20:26.641+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23626 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:20:31.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.640+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338004e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:20:31.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.640+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338004e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:31.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.640+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338004e30 secure :-1 s=READY pgs=8178 cs=0 l=1 rev1=1 rx=0x7f833404e040 tx=0x7f833404b2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:31.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.640+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338004e30 secure :-1 s=READY pgs=8178 cs=0 l=1 rev1=1 rx=0x7f833404e040 tx=0x7f833404b2d0).stop 2021-03-09T10:20:31.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.640+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338004e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:31.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.640+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338004e30 unknown :-1 s=CLOSED pgs=8178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:31.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.640+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:31.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.640+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:31.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.640+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:31.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.644+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e890 con 0x7f8324003400 2021-03-09T10:20:31.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.644+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338071d10 2021-03-09T10:20:31.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.644+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069940 con 0x7f8324005e00 2021-03-09T10:20:31.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.644+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:20:31.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.644+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:20:31.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.644+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:31.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.644+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:31.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.644+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:31.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.644+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:31.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.644+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:31.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.644+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:31.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.644+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:31.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.644+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380062d0 con 0x7f8338071d10 2021-03-09T10:20:31.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.644+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8188 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:31.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0a1b40 con 0x7f8338071d10 2021-03-09T10:20:31.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0a16e0 con 0x7f8338071d10 2021-03-09T10:20:31.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:31.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c330 con 0x7f8338071d10 2021-03-09T10:20:32.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:20:31 smithi146 bash[13136]: debug 2021-03-09T10:20:31.644+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23653 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:20:36.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:20:36.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:36.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8188 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:36.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8188 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c00c2c0).stop 2021-03-09T10:20:36.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:36.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:36.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:36.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:36.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338013df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:36.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073bb0 con 0x7f8324003400 2021-03-09T10:20:36.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a0d0 con 0x7f8338069320 2021-03-09T10:20:36.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c290 con 0x7f8324005e00 2021-03-09T10:20:36.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:20:36.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:36.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:20:36.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338013df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:36.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:36.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:36.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:36.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:36.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:36.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338069320 2021-03-09T10:20:36.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.648+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338013df0 secure :-1 s=READY pgs=8196 cs=0 l=1 rev1=1 rx=0x7f8338005d80 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:36.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01a270 con 0x7f8338069320 2021-03-09T10:20:36.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f8338069320 2021-03-09T10:20:36.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:36.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01a270 con 0x7f8338069320 2021-03-09T10:20:37.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:20:36 smithi146 bash[13136]: debug 2021-03-09T10:20:36.648+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23678 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:20:41.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.648+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338013df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:20:41.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.648+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338013df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:41.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.648+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338013df0 secure :-1 s=READY pgs=8196 cs=0 l=1 rev1=1 rx=0x7f8338005d80 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:41.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.648+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338013df0 secure :-1 s=READY pgs=8196 cs=0 l=1 rev1=1 rx=0x7f8338005d80 tx=0x7f833c00c2c0).stop 2021-03-09T10:20:41.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338013df0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:41.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.648+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338013df0 unknown :-1 s=CLOSED pgs=8196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:41.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:41.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:41.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:20:41.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:20:41.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:41.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:41.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:41.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025490 con 0x7f8324003400 2021-03-09T10:20:41.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8338071d10 2021-03-09T10:20:41.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dce0 con 0x7f8324005e00 2021-03-09T10:20:41.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:41.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:41.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:41.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:41.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:41.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338061fc0 con 0x7f8338071d10 2021-03-09T10:20:41.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.652+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8206 cs=0 l=1 rev1=1 rx=0x7f833c01e040 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:41.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0a1420 con 0x7f8338071d10 2021-03-09T10:20:41.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0a1420 con 0x7f8338071d10 2021-03-09T10:20:41.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:41.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01a2e0 con 0x7f8338071d10 2021-03-09T10:20:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:20:41 smithi146 bash[13136]: debug 2021-03-09T10:20:41.652+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23704 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:20:46.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:20:46.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:46.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8206 cs=0 l=1 rev1=1 rx=0x7f833c01e040 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:46.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8206 cs=0 l=1 rev1=1 rx=0x7f833c01e040 tx=0x7f833c00c2c0).stop 2021-03-09T10:20:46.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:46.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:46.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:46.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:46.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:46.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069940 con 0x7f8324003400 2021-03-09T10:20:46.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016b30 con 0x7f8338069320 2021-03-09T10:20:46.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025490 con 0x7f8324005e00 2021-03-09T10:20:46.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:20:46.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:46.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:20:46.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:46.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:46.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:46.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:46.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:46.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:46.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065440 con 0x7f8338069320 2021-03-09T10:20:46.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.656+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801af20 secure :-1 s=READY pgs=8213 cs=0 l=1 rev1=1 rx=0x7f833404fb00 tx=0x7f833404b2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:46.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338069320 2021-03-09T10:20:46.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338069320 2021-03-09T10:20:46.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:46.668+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403a1f0 con 0x7f8338069320 2021-03-09T10:20:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:20:46 smithi146 bash[13136]: debug 2021-03-09T10:20:46.656+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23730 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:20:51.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:20:51.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:51.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801af20 secure :-1 s=READY pgs=8213 cs=0 l=1 rev1=1 rx=0x7f833404fb00 tx=0x7f833404b2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:51.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801af20 secure :-1 s=READY pgs=8213 cs=0 l=1 rev1=1 rx=0x7f833404fb00 tx=0x7f833404b2d0).stop 2021-03-09T10:20:51.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801af20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:51.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801af20 unknown :-1 s=CLOSED pgs=8213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:51.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:51.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:51.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:51.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324003400 2021-03-09T10:20:51.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380030d0 con 0x7f8338071d10 2021-03-09T10:20:51.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324005e00 2021-03-09T10:20:51.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:20:51.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:51.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:20:51.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:51.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:51.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380062d0 con 0x7f8338071d10 2021-03-09T10:20:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8221 cs=0 l=1 rev1=1 rx=0x7f833800a690 tx=0x7f834404c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:51.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T10:20:51.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.664+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T10:20:51.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:51.668+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aaf0 con 0x7f8338071d10 2021-03-09T10:20:52.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:20:51 smithi146 bash[13136]: debug 2021-03-09T10:20:51.660+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23756 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:20:56.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:20:56.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:56.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8221 cs=0 l=1 rev1=1 rx=0x7f833800a690 tx=0x7f834404c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:56.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8221 cs=0 l=1 rev1=1 rx=0x7f833800a690 tx=0x7f834404c330).stop 2021-03-09T10:20:56.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:56.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:56.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:56.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:56.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:56.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073900 con 0x7f8324003400 2021-03-09T10:20:56.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e0e0 con 0x7f8338069320 2021-03-09T10:20:56.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a90 con 0x7f8324005e00 2021-03-09T10:20:56.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:20:56.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:56.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:20:56.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:56.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:56.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:56.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:56.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:56.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:56.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338069320 2021-03-09T10:20:56.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 secure :-1 s=READY pgs=8230 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834402bec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:56.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:20:56.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344017640 con 0x7f8338069320 2021-03-09T10:20:56.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:20:56.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ac90 con 0x7f8338069320 2021-03-09T10:20:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:20:56 smithi146 bash[13136]: debug 2021-03-09T10:20:56.664+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23782 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:21:01.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833806ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:21:01.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833806ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:01.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 secure :-1 s=READY pgs=8230 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834402bec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:01.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 secure :-1 s=READY pgs=8230 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834402bec0).stop 2021-03-09T10:21:01.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833806ce20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:01.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 unknown :-1 s=CLOSED pgs=8230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:01.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:01.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:01.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:01.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380675d0 con 0x7f8324003400 2021-03-09T10:21:01.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338071d10 2021-03-09T10:21:01.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a130 con 0x7f8324005e00 2021-03-09T10:21:01.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:21:01.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:21:01.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:01.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:01.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:01.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:01.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:01.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.675+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:01.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.675+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:01.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.675+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380131a0 con 0x7f8338071d10 2021-03-09T10:21:01.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.675+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8239 cs=0 l=1 rev1=1 rx=0x7f83440134c0 tx=0x7f834402bec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:01.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T10:21:01.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f8338071d10 2021-03-09T10:21:01.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:01.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440171c0 con 0x7f8338071d10 2021-03-09T10:21:02.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:21:01 smithi146 bash[13136]: debug 2021-03-09T10:21:01.672+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23809 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:21:06.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.675+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:21:06.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.675+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:06.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.675+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8239 cs=0 l=1 rev1=1 rx=0x7f83440134c0 tx=0x7f834402bec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:06.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.675+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8239 cs=0 l=1 rev1=1 rx=0x7f83440134c0 tx=0x7f834402bec0).stop 2021-03-09T10:21:06.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:06.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.675+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:06.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.675+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:06.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.675+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338011970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:06.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.675+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:06.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018a90 con 0x7f8324003400 2021-03-09T10:21:06.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380059b0 con 0x7f8338069320 2021-03-09T10:21:06.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e0e0 con 0x7f8324005e00 2021-03-09T10:21:06.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.675+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:21:06.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.675+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:06.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.675+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:21:06.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.679+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:06.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.679+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338011970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:06.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.679+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:06.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.679+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:06.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.679+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:06.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.679+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:06.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.679+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801c090 con 0x7f8338069320 2021-03-09T10:21:06.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.679+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338011970 secure :-1 s=READY pgs=8247 cs=0 l=1 rev1=1 rx=0x7f83380189c0 tx=0x7f834402bec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:06.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404ed80 con 0x7f8338069320 2021-03-09T10:21:06.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f8338069320 2021-03-09T10:21:06.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:06.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440144a0 con 0x7f8338069320 2021-03-09T10:21:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:21:06 smithi146 bash[13136]: debug 2021-03-09T10:21:06.676+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23834 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:21:11.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.679+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338011970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:21:11.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.679+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338011970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:11.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.679+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338011970 secure :-1 s=READY pgs=8247 cs=0 l=1 rev1=1 rx=0x7f83380189c0 tx=0x7f834402bec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:11.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.679+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338011970 secure :-1 s=READY pgs=8247 cs=0 l=1 rev1=1 rx=0x7f83380189c0 tx=0x7f834402bec0).stop 2021-03-09T10:21:11.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338011970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:11.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.679+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338011970 unknown :-1 s=CLOSED pgs=8247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:11.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.679+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:11.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.679+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:11.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.683+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:11.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8324003400 2021-03-09T10:21:11.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a150 con 0x7f8338071d10 2021-03-09T10:21:11.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:21:11.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.683+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:21:11.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.683+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:11.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.683+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:21:11.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.683+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:11.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.683+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:11.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.683+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:11.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.683+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:11.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.683+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:11.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.683+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:11.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.683+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c290 con 0x7f8338071d10 2021-03-09T10:21:11.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.683+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8256 cs=0 l=1 rev1=1 rx=0x7f8334034050 tx=0x7f833404b2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:11.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.687+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401f440 con 0x7f8338071d10 2021-03-09T10:21:11.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.687+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402da60 con 0x7f8338071d10 2021-03-09T10:21:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:11.691+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404b380 con 0x7f8338071d10 2021-03-09T10:21:12.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:21:11 smithi146 bash[13136]: debug 2021-03-09T10:21:11.680+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23860 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:21:16.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.683+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:21:16.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.683+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:16.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.683+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8256 cs=0 l=1 rev1=1 rx=0x7f8334034050 tx=0x7f833404b2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:16.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.683+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8256 cs=0 l=1 rev1=1 rx=0x7f8334034050 tx=0x7f833404b2d0).stop 2021-03-09T10:21:16.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:16.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.683+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:16.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.683+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380174c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:16.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.683+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:16.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.683+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:16.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e0e0 con 0x7f8324003400 2021-03-09T10:21:16.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010640 con 0x7f8338069320 2021-03-09T10:21:16.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064cb0 con 0x7f8324005e00 2021-03-09T10:21:16.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.687+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:21:16.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.687+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:21:16.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.687+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:16.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.687+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:16.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.687+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380174c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:16.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.687+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:16.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.687+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:16.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.687+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:16.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.687+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:16.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.687+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338069320 2021-03-09T10:21:16.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.687+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380174c0 secure :-1 s=READY pgs=8265 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833404b2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:16.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.691+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404b6c0 con 0x7f8338069320 2021-03-09T10:21:16.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.691+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334052450 con 0x7f8338069320 2021-03-09T10:21:16.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:16.695+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403ad60 con 0x7f8338069320 2021-03-09T10:21:17.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:21:16 smithi146 bash[13136]: debug 2021-03-09T10:21:16.684+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23886 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:21:21.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380174c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:21:21.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380174c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:21.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380174c0 secure :-1 s=READY pgs=8265 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833404b2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:21.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380174c0 secure :-1 s=READY pgs=8265 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833404b2d0).stop 2021-03-09T10:21:21.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380174c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:21.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380174c0 unknown :-1 s=CLOSED pgs=8265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:21.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:21.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:21.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:21.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324003400 2021-03-09T10:21:21.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380084d0 con 0x7f8338071d10 2021-03-09T10:21:21.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380075d0 con 0x7f8324005e00 2021-03-09T10:21:21.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:21:21.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:21.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:21:21.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:21.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.691+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:21.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.695+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:21.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.695+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:21.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.695+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:21.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.695+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:21.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.695+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e840 con 0x7f8338071d10 2021-03-09T10:21:21.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.695+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8276 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834402bec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:21.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.695+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404ed80 con 0x7f8338071d10 2021-03-09T10:21:21.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.699+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404ed80 con 0x7f8338071d10 2021-03-09T10:21:21.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:21.699+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344057330 con 0x7f8338071d10 2021-03-09T10:21:22.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:21:21 smithi146 bash[13136]: debug 2021-03-09T10:21:21.688+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23912 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:21:26.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:21:26.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:26.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8276 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834402bec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:26.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8276 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834402bec0).stop 2021-03-09T10:21:26.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:26.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:26.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:26.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:26.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:26.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ca0 con 0x7f8324003400 2021-03-09T10:21:26.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8338069320 2021-03-09T10:21:26.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024bb0 con 0x7f8324005e00 2021-03-09T10:21:26.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:21:26.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:26.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:21:26.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:26.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.695+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:26.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.699+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:26.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.699+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:26.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.699+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:26.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.699+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:26.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.699+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380172b0 con 0x7f8338069320 2021-03-09T10:21:26.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.699+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 secure :-1 s=READY pgs=8284 cs=0 l=1 rev1=1 rx=0x7f83380107b0 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:26.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.699+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00c330 con 0x7f8338069320 2021-03-09T10:21:26.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.699+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0553a0 con 0x7f8338069320 2021-03-09T10:21:26.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:26.703+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338069320 2021-03-09T10:21:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:21:26 smithi146 bash[13136]: debug 2021-03-09T10:21:26.696+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23938 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:21:31.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833806ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:21:31.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833806ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:31.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 secure :-1 s=READY pgs=8284 cs=0 l=1 rev1=1 rx=0x7f83380107b0 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:31.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 secure :-1 s=READY pgs=8284 cs=0 l=1 rev1=1 rx=0x7f83380107b0 tx=0x7f833c00c2c0).stop 2021-03-09T10:21:31.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833806ce20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:31.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 unknown :-1 s=CLOSED pgs=8284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:31.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:31.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:31.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:31.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ca0 con 0x7f8324003400 2021-03-09T10:21:31.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010350 con 0x7f8338071d10 2021-03-09T10:21:31.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066e70 con 0x7f8324005e00 2021-03-09T10:21:31.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:21:31.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:31.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:21:31.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:31.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:31.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:31.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:31.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:31.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:31.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.699+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c290 con 0x7f8338071d10 2021-03-09T10:21:31.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.703+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8291 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404ebb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:31.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.703+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344078e20 con 0x7f8338071d10 2021-03-09T10:21:31.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.703+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f8338071d10 2021-03-09T10:21:31.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:31.707+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404ec20 con 0x7f8338071d10 2021-03-09T10:21:32.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:21:31 smithi146 bash[13136]: debug 2021-03-09T10:21:31.700+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23964 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:21:36.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:21:36.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:36.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8291 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404ebb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:36.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8291 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834404ebb0).stop 2021-03-09T10:21:36.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:36.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:36.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:36.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:36.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:36.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d0d0 con 0x7f8324003400 2021-03-09T10:21:36.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074330 con 0x7f8338069320 2021-03-09T10:21:36.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ca0 con 0x7f8324005e00 2021-03-09T10:21:36.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:36.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:21:36.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:21:36.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:36.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.703+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:36.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.707+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:36.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.707+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:36.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.707+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:36.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.707+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:36.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.707+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a150 con 0x7f8338069320 2021-03-09T10:21:36.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.707+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 secure :-1 s=READY pgs=8300 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404ebb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:36.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.707+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344078e20 con 0x7f8338069320 2021-03-09T10:21:36.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.707+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405be70 con 0x7f8338069320 2021-03-09T10:21:36.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:36.711+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f8338069320 2021-03-09T10:21:37.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:21:36 smithi146 bash[13136]: debug 2021-03-09T10:21:36.704+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 23990 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:21:41.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.707+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:21:41.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.707+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:41.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.707+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 secure :-1 s=READY pgs=8300 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404ebb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:41.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.707+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 secure :-1 s=READY pgs=8300 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404ebb0).stop 2021-03-09T10:21:41.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.707+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338019650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:41.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.707+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 unknown :-1 s=CLOSED pgs=8300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:41.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.707+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:41.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.707+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:41.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.707+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:41.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.707+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324003400 2021-03-09T10:21:41.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.707+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ca0 con 0x7f8338071d10 2021-03-09T10:21:41.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.707+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008050 con 0x7f8324005e00 2021-03-09T10:21:41.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.715+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:21:41.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.715+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:41.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.715+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:21:41.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.715+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:41.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.715+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:41.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.719+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:41.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.719+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:41.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.719+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:41.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.719+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:41.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.719+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801c090 con 0x7f8338071d10 2021-03-09T10:21:41.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.719+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8313 cs=0 l=1 rev1=1 rx=0x7f83440117d0 tx=0x7f834404ebb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:41.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.719+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T10:21:41.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.719+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405be70 con 0x7f8338071d10 2021-03-09T10:21:41.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:41.719+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440470f0 con 0x7f8338071d10 2021-03-09T10:21:42.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:21:41 smithi146 bash[13136]: debug 2021-03-09T10:21:41.707+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24016 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:21:46.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:21:46.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:46.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8313 cs=0 l=1 rev1=1 rx=0x7f83440117d0 tx=0x7f834404ebb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:46.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8313 cs=0 l=1 rev1=1 rx=0x7f83440117d0 tx=0x7f834404ebb0).stop 2021-03-09T10:21:46.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:46.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:46.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:46.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:46.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:46.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ca0 con 0x7f8324003400 2021-03-09T10:21:46.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8338069320 2021-03-09T10:21:46.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324005e00 2021-03-09T10:21:46.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:21:46.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:21:46.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:46.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:46.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:46.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:46.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:46.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:46.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.710+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:46.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.714+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338069320 2021-03-09T10:21:46.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.714+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 secure :-1 s=READY pgs=8317 cs=0 l=1 rev1=1 rx=0x7f8334056a40 tx=0x7f833404b2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:46.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.714+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400d310 con 0x7f8338069320 2021-03-09T10:21:46.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.714+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334031130 con 0x7f8338069320 2021-03-09T10:21:46.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:46.718+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404ecf0 con 0x7f8338069320 2021-03-09T10:21:47.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:21:46 smithi146 bash[13136]: debug 2021-03-09T10:21:46.711+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24042 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:21:51.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:21:51.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:51.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 secure :-1 s=READY pgs=8317 cs=0 l=1 rev1=1 rx=0x7f8334056a40 tx=0x7f833404b2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:51.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 secure :-1 s=READY pgs=8317 cs=0 l=1 rev1=1 rx=0x7f8334056a40 tx=0x7f833404b2d0).stop 2021-03-09T10:21:51.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338019650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:51.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 unknown :-1 s=CLOSED pgs=8317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:51.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:51.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:51.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:51.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:21:51.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d400 con 0x7f8324003400 2021-03-09T10:21:51.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:51.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ca0 con 0x7f8338071d10 2021-03-09T10:21:51.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324005e00 2021-03-09T10:21:51.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:21:51.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:51.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.714+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:51.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.718+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:51.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.718+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:51.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.718+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:51.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.718+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:51.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.718+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380217d0 con 0x7f8338071d10 2021-03-09T10:21:51.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.718+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8327 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f83440134d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:51.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.718+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T10:21:51.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.718+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033030 con 0x7f8338071d10 2021-03-09T10:21:51.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:51.722+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405bd70 con 0x7f8338071d10 2021-03-09T10:21:52.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:21:51 smithi146 bash[13136]: debug 2021-03-09T10:21:51.715+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24068 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:21:56.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:21:56.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:56.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8327 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f83440134d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:56.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8327 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f83440134d0).stop 2021-03-09T10:21:56.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:56.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:56.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:56.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:56.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:56.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324003400 2021-03-09T10:21:56.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064cb0 con 0x7f8338069320 2021-03-09T10:21:56.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:21:56.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:21:56.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:56.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:21:56.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:56.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.718+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:56.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.722+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:56.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.722+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:56.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.722+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:56.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.722+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:56.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.722+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806ca50 con 0x7f8338069320 2021-03-09T10:21:56.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.722+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380091b0 secure :-1 s=READY pgs=8334 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f833404b2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:56.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.726+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833403bb10 con 0x7f8338069320 2021-03-09T10:21:56.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.726+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007c10 con 0x7f8338069320 2021-03-09T10:21:56.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:21:56.730+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334046aa0 con 0x7f8338069320 2021-03-09T10:21:57.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:21:56 smithi146 bash[13136]: debug 2021-03-09T10:21:56.723+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24094 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:22:01.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.722+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:22:01.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.722+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:01.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.722+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380091b0 secure :-1 s=READY pgs=8334 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f833404b2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:01.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.722+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380091b0 secure :-1 s=READY pgs=8334 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f833404b2d0).stop 2021-03-09T10:22:01.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.722+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380091b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:01.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.722+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380091b0 unknown :-1 s=CLOSED pgs=8334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:01.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.722+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:01.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.722+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:01.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.722+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:01.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.722+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028cb0 con 0x7f8324003400 2021-03-09T10:22:01.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.722+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a080 con 0x7f8338071d10 2021-03-09T10:22:01.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.722+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005ca0 con 0x7f8324005e00 2021-03-09T10:22:01.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.722+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:22:01.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.722+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:01.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.722+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:22:01.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.722+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:01.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.726+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:01.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.726+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:01.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.726+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:01.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.726+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:01.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.726+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:01.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.726+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e840 con 0x7f8338071d10 2021-03-09T10:22:01.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.730+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8343 cs=0 l=1 rev1=1 rx=0x7f833406da20 tx=0x7f833404b2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:01.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.730+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340261b0 con 0x7f8338071d10 2021-03-09T10:22:01.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.730+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007c10 con 0x7f8338071d10 2021-03-09T10:22:01.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:01.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403b290 con 0x7f8338071d10 2021-03-09T10:22:02.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:22:01 smithi146 bash[13136]: debug 2021-03-09T10:22:01.723+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24120 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:22:06.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:22:06.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:06.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8343 cs=0 l=1 rev1=1 rx=0x7f833406da20 tx=0x7f833404b2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:06.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8343 cs=0 l=1 rev1=1 rx=0x7f833406da20 tx=0x7f833404b2d0).stop 2021-03-09T10:22:06.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:06.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:06.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:06.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:06.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:06.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021820 con 0x7f8324003400 2021-03-09T10:22:06.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8338069320 2021-03-09T10:22:06.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380084d0 con 0x7f8324005e00 2021-03-09T10:22:06.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:22:06.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:06.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:22:06.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:06.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.726+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:06.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.730+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:06.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.730+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:06.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.730+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:06.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.730+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:06.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.730+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338016f20 con 0x7f8338069320 2021-03-09T10:22:06.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.730+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 secure :-1 s=READY pgs=8351 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:06.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.730+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c035370 con 0x7f8338069320 2021-03-09T10:22:06.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.730+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f8338069320 2021-03-09T10:22:06.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:06.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f8338069320 2021-03-09T10:22:07.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:22:06 smithi146 bash[13136]: debug 2021-03-09T10:22:06.731+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24146 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:22:11.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833806ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:22:11.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833806ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:11.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 secure :-1 s=READY pgs=8351 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:11.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 secure :-1 s=READY pgs=8351 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c00c2c0).stop 2021-03-09T10:22:11.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833806ce20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:11.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833806ce20 unknown :-1 s=CLOSED pgs=8351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:11.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:11.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:11.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:11.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073d60 con 0x7f8324003400 2021-03-09T10:22:11.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069b40 con 0x7f8338071d10 2021-03-09T10:22:11.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028cb0 con 0x7f8324005e00 2021-03-09T10:22:11.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:22:11.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:11.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:22:11.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:11.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:11.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:11.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:11.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:11.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:11.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338071d10 2021-03-09T10:22:11.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.734+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8360 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:11.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0412e0 con 0x7f8338071d10 2021-03-09T10:22:11.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0416f0 con 0x7f8338071d10 2021-03-09T10:22:11.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:11.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c020bb0 con 0x7f8338071d10 2021-03-09T10:22:12.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:22:11 smithi146 bash[13136]: debug 2021-03-09T10:22:11.735+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24172 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:22:16.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.734+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:22:16.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.734+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:16.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.734+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8360 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:16.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.734+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8360 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c00c2c0).stop 2021-03-09T10:22:16.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:16.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.734+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:16.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.734+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:16.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.734+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:16.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.734+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:16.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019f30 con 0x7f8324003400 2021-03-09T10:22:16.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d980 con 0x7f8338069320 2021-03-09T10:22:16.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324005e00 2021-03-09T10:22:16.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.734+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:22:16.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.738+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:16.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.738+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:16.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.738+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:22:16.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.738+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:16.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.738+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:16.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.738+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:16.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.738+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:16.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.738+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:16.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.738+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338013fe0 con 0x7f8338069320 2021-03-09T10:22:16.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.738+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020450 secure :-1 s=READY pgs=8369 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:16.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f8338069320 2021-03-09T10:22:16.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033050 con 0x7f8338069320 2021-03-09T10:22:16.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:16.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:22:17.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:22:16 smithi146 bash[13136]: debug 2021-03-09T10:22:16.739+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24198 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:22:21.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338020450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:22:21.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338020450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:21.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020450 secure :-1 s=READY pgs=8369 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:21.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020450 secure :-1 s=READY pgs=8369 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344004c60).stop 2021-03-09T10:22:21.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338020450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020450 unknown :-1 s=CLOSED pgs=8369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:22:21.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380084d0 con 0x7f8324003400 2021-03-09T10:22:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018620 con 0x7f8338071d10 2021-03-09T10:22:21.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a080 con 0x7f8324005e00 2021-03-09T10:22:21.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:22:21.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:21.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:21.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:21.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:21.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:21.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:21.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.742+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005ca0 con 0x7f8338071d10 2021-03-09T10:22:21.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.746+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8377 cs=0 l=1 rev1=1 rx=0x7f833c045510 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:21.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c036700 con 0x7f8338071d10 2021-03-09T10:22:21.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c036930 con 0x7f8338071d10 2021-03-09T10:22:21.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:21.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338071d10 2021-03-09T10:22:22.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:22:21 smithi146 bash[13136]: debug 2021-03-09T10:22:21.743+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24224 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:22:26.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:22:26.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:26.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8377 cs=0 l=1 rev1=1 rx=0x7f833c045510 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:26.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8377 cs=0 l=1 rev1=1 rx=0x7f833c045510 tx=0x7f833c00c2c0).stop 2021-03-09T10:22:26.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:26.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:26.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:26.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:22:26.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:26.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338066350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:26.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:26.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324003400 2021-03-09T10:22:26.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b800 con 0x7f8338069320 2021-03-09T10:22:26.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806f020 con 0x7f8324005e00 2021-03-09T10:22:26.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338066350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:26.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:22:26.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:26.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:26.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:26.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:26.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:26.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380262a0 con 0x7f8338069320 2021-03-09T10:22:26.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.746+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338066350 secure :-1 s=READY pgs=8386 cs=0 l=1 rev1=1 rx=0x7f8344066b10 tx=0x7f8344004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:26.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344066eb0 con 0x7f8338069320 2021-03-09T10:22:26.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405edd0 con 0x7f8338069320 2021-03-09T10:22:26.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:26.754+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:22:27.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:22:26 smithi146 bash[13136]: debug 2021-03-09T10:22:26.747+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24250 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:22:31.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338066350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:22:31.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338066350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:31.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338066350 secure :-1 s=READY pgs=8386 cs=0 l=1 rev1=1 rx=0x7f8344066b10 tx=0x7f8344004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:31.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338066350 secure :-1 s=READY pgs=8386 cs=0 l=1 rev1=1 rx=0x7f8344066b10 tx=0x7f8344004c60).stop 2021-03-09T10:22:31.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338066350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:31.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338066350 unknown :-1 s=CLOSED pgs=8386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:31.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:31.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:31.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:31.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:31.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:22:31.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:31.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:22:31.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:31.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066e70 con 0x7f8324003400 2021-03-09T10:22:31.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806f020 con 0x7f8338071d10 2021-03-09T10:22:31.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072660 con 0x7f8324005e00 2021-03-09T10:22:31.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:31.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:31.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:31.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:31.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338071d10 2021-03-09T10:22:31.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.750+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8395 cs=0 l=1 rev1=1 rx=0x7f8334004f90 tx=0x7f833404b2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:31.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.754+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404ead0 con 0x7f8338071d10 2021-03-09T10:22:31.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.754+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334060920 con 0x7f8338071d10 2021-03-09T10:22:31.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:31.758+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833406d1f0 con 0x7f8338071d10 2021-03-09T10:22:32.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:22:31 smithi146 bash[13136]: debug 2021-03-09T10:22:31.751+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24276 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:22:36.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:22:36.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:36.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8395 cs=0 l=1 rev1=1 rx=0x7f8334004f90 tx=0x7f833404b2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:36.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8395 cs=0 l=1 rev1=1 rx=0x7f8334004f90 tx=0x7f833404b2d0).stop 2021-03-09T10:22:36.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:36.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:36.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:36.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:36.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:36.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028cb0 con 0x7f8324003400 2021-03-09T10:22:36.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e880 con 0x7f8338069320 2021-03-09T10:22:36.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015760 con 0x7f8324005e00 2021-03-09T10:22:36.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:22:36.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:36.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:36.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:22:36.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:36.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:36.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:36.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:36.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:36.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278c0 con 0x7f8338069320 2021-03-09T10:22:36.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.753+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 secure :-1 s=READY pgs=8403 cs=0 l=1 rev1=1 rx=0x7f8334004f90 tx=0x7f833404b2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:36.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.757+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833403b290 con 0x7f8338069320 2021-03-09T10:22:36.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.757+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338069320 2021-03-09T10:22:36.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:36.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334064290 con 0x7f8338069320 2021-03-09T10:22:37.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:22:36 smithi146 bash[13136]: debug 2021-03-09T10:22:36.755+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24301 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:22:41.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:22:41.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:41.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 secure :-1 s=READY pgs=8403 cs=0 l=1 rev1=1 rx=0x7f8334004f90 tx=0x7f833404b2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:41.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 secure :-1 s=READY pgs=8403 cs=0 l=1 rev1=1 rx=0x7f8334004f90 tx=0x7f833404b2d0).stop 2021-03-09T10:22:41.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:41.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 unknown :-1 s=CLOSED pgs=8403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:41.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:41.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:41.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:41.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fd90 con 0x7f8324003400 2021-03-09T10:22:41.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:22:41.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028cb0 con 0x7f8338071d10 2021-03-09T10:22:41.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380172f0 con 0x7f8324005e00 2021-03-09T10:22:41.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:41.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:41.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:22:41.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:41.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:41.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:41.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:41.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:41.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.757+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ad60 con 0x7f8338071d10 2021-03-09T10:22:41.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.761+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8412 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:41.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00c400 con 0x7f8338071d10 2021-03-09T10:22:41.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c010350 con 0x7f8338071d10 2021-03-09T10:22:41.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:41.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0041c0 con 0x7f8338071d10 2021-03-09T10:22:42.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:22:41 smithi146 bash[13136]: debug 2021-03-09T10:22:41.759+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24328 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:22:46.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.761+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:22:46.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.761+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:46.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.761+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8412 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:46.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.761+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8412 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c00c2c0).stop 2021-03-09T10:22:46.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:46.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.761+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:46.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.761+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:46.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.761+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:22:46.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.761+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:46.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.761+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:46.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.761+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:46.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fd90 con 0x7f8324003400 2021-03-09T10:22:46.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380075f0 con 0x7f8338069320 2021-03-09T10:22:46.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062970 con 0x7f8324005e00 2021-03-09T10:22:46.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.761+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:46.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.765+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:22:46.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.765+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:46.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:46.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:46.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:46.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:46.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338069320 2021-03-09T10:22:46.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802d530 secure :-1 s=READY pgs=8422 cs=0 l=1 rev1=1 rx=0x7f834403be40 tx=0x7f8344004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:46.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.769+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:22:46.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.769+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:22:46.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:46.773+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:22:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:22:46 smithi146 bash[13136]: debug 2021-03-09T10:22:46.763+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24353 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:22:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:22:51.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:51.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802d530 secure :-1 s=READY pgs=8422 cs=0 l=1 rev1=1 rx=0x7f834403be40 tx=0x7f8344004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802d530 secure :-1 s=READY pgs=8422 cs=0 l=1 rev1=1 rx=0x7f834403be40 tx=0x7f8344004c60).stop 2021-03-09T10:22:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802d530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802d530 unknown :-1 s=CLOSED pgs=8422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:22:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:22:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380259a0 con 0x7f8324003400 2021-03-09T10:22:51.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028cb0 con 0x7f8338071d10 2021-03-09T10:22:51.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fd90 con 0x7f8324005e00 2021-03-09T10:22:51.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.765+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:51.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.769+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:51.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.769+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:51.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.769+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:51.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.769+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:51.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.769+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278c0 con 0x7f8338071d10 2021-03-09T10:22:51.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.769+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8429 cs=0 l=1 rev1=1 rx=0x7f8338018530 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:51.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.773+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04d240 con 0x7f8338071d10 2021-03-09T10:22:51.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.773+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c010350 con 0x7f8338071d10 2021-03-09T10:22:51.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:51.777+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054cd0 con 0x7f8338071d10 2021-03-09T10:22:52.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:22:51 smithi146 bash[13136]: debug 2021-03-09T10:22:51.770+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24380 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:22:56.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:22:56.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:56.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8429 cs=0 l=1 rev1=1 rx=0x7f8338018530 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:56.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8429 cs=0 l=1 rev1=1 rx=0x7f8338018530 tx=0x7f833c00c2c0).stop 2021-03-09T10:22:56.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:56.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:56.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:56.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:22:56.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:56.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338063080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:56.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:56.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027780 con 0x7f8324003400 2021-03-09T10:22:56.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024bb0 con 0x7f8338069320 2021-03-09T10:22:56.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380651a0 con 0x7f8324005e00 2021-03-09T10:22:56.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338063080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:56.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:22:56.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:56.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:56.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:56.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:56.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:56.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338002fe0 con 0x7f8338069320 2021-03-09T10:22:56.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.773+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338063080 secure :-1 s=READY pgs=8438 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:56.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.777+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344010250 con 0x7f8338069320 2021-03-09T10:22:56.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.777+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834403be40 con 0x7f8338069320 2021-03-09T10:22:56.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:22:56.781+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:22:57.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:22:56 smithi146 bash[13136]: debug 2021-03-09T10:22:56.774+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24405 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:23:01.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.773+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:23:01.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.773+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:01.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.773+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338063080 secure :-1 s=READY pgs=8438 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:01.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.773+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338063080 secure :-1 s=READY pgs=8438 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344004c60).stop 2021-03-09T10:23:01.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.773+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338063080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:01.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.773+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338063080 unknown :-1 s=CLOSED pgs=8438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:01.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.773+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:01.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.777+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:01.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.777+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.777+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380651a0 con 0x7f8324003400 2021-03-09T10:23:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.777+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012290 con 0x7f8338071d10 2021-03-09T10:23:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.777+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324005e00 2021-03-09T10:23:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.777+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:23:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.777+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:01.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.777+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:23:01.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.777+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:01.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.777+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:01.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.777+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:01.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.777+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:01.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.777+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:01.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.777+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:01.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.777+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338071d10 2021-03-09T10:23:01.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.777+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8447 cs=0 l=1 rev1=1 rx=0x7f8334000e10 tx=0x7f833404b2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:01.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.781+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833402dae0 con 0x7f8338071d10 2021-03-09T10:23:01.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.781+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338071d10 2021-03-09T10:23:01.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:01.785+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403b360 con 0x7f8338071d10 2021-03-09T10:23:02.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:23:01 smithi146 bash[13136]: debug 2021-03-09T10:23:01.778+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24432 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:23:06.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:23:06.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:06.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8447 cs=0 l=1 rev1=1 rx=0x7f8334000e10 tx=0x7f833404b2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:06.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8447 cs=0 l=1 rev1=1 rx=0x7f8334000e10 tx=0x7f833404b2d0).stop 2021-03-09T10:23:06.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:06.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:06.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:06.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:06.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:06.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024bb0 con 0x7f8324003400 2021-03-09T10:23:06.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380651a0 con 0x7f8338069320 2021-03-09T10:23:06.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380265a0 con 0x7f8324005e00 2021-03-09T10:23:06.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:23:06.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:06.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:23:06.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:06.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:06.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:06.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:06.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:06.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:06.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278c0 con 0x7f8338069320 2021-03-09T10:23:06.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.781+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802d530 secure :-1 s=READY pgs=8455 cs=0 l=1 rev1=1 rx=0x7f8338069af0 tx=0x7f8344004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:06.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.785+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344066e80 con 0x7f8338069320 2021-03-09T10:23:06.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.785+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344066e80 con 0x7f8338069320 2021-03-09T10:23:06.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:06.789+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440470f0 con 0x7f8338069320 2021-03-09T10:23:07.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:23:06 smithi146 bash[13136]: debug 2021-03-09T10:23:06.782+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24456 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:23:11.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:23:11.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:11.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802d530 secure :-1 s=READY pgs=8455 cs=0 l=1 rev1=1 rx=0x7f8338069af0 tx=0x7f8344004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:11.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802d530 secure :-1 s=READY pgs=8455 cs=0 l=1 rev1=1 rx=0x7f8338069af0 tx=0x7f8344004c60).stop 2021-03-09T10:23:11.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802d530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:11.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802d530 unknown :-1 s=CLOSED pgs=8455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:11.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:11.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:11.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:11.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801fda0 con 0x7f8324003400 2021-03-09T10:23:11.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8338071d10 2021-03-09T10:23:11.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380651a0 con 0x7f8324005e00 2021-03-09T10:23:11.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:11.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:23:11.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:23:11.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:11.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.785+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:11.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.789+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:11.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.789+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:11.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.789+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:11.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.789+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:11.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.789+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338071d10 2021-03-09T10:23:11.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.789+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8467 cs=0 l=1 rev1=1 rx=0x7f833400bbf0 tx=0x7f833404b2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:11.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.789+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338071d10 2021-03-09T10:23:11.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.789+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402da60 con 0x7f8338071d10 2021-03-09T10:23:11.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:11.793+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404b4b0 con 0x7f8338071d10 2021-03-09T10:23:12.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:23:11 smithi146 bash[13136]: debug 2021-03-09T10:23:11.786+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24484 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:23:16.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:23:16.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:16.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8467 cs=0 l=1 rev1=1 rx=0x7f833400bbf0 tx=0x7f833404b2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:16.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8467 cs=0 l=1 rev1=1 rx=0x7f833400bbf0 tx=0x7f833404b2d0).stop 2021-03-09T10:23:16.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:16.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:16.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:16.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380637b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:16.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:16.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:23:16.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b0f0 con 0x7f8324003400 2021-03-09T10:23:16.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:16.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380651a0 con 0x7f8338069320 2021-03-09T10:23:16.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019f30 con 0x7f8324005e00 2021-03-09T10:23:16.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:23:16.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:16.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:16.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:16.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:16.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:16.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:16.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.789+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380757a0 con 0x7f8338069320 2021-03-09T10:23:16.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.793+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380637b0 secure :-1 s=READY pgs=8473 cs=0 l=1 rev1=1 rx=0x7f83440194f0 tx=0x7f8344004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:16.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.793+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403c280 con 0x7f8338069320 2021-03-09T10:23:16.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.793+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:23:16.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:16.797+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:23:17.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:23:16 smithi146 bash[13136]: debug 2021-03-09T10:23:16.790+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24508 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:23:21.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380637b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:23:21.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380637b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:21.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380637b0 secure :-1 s=READY pgs=8473 cs=0 l=1 rev1=1 rx=0x7f83440194f0 tx=0x7f8344004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:21.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380637b0 secure :-1 s=READY pgs=8473 cs=0 l=1 rev1=1 rx=0x7f83440194f0 tx=0x7f8344004c60).stop 2021-03-09T10:23:21.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380637b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:21.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380637b0 unknown :-1 s=CLOSED pgs=8473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:21.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:21.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:23:21.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:21.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:21.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:21.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338006090 con 0x7f8324003400 2021-03-09T10:23:21.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801fda0 con 0x7f8338071d10 2021-03-09T10:23:21.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380651a0 con 0x7f8324005e00 2021-03-09T10:23:21.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:23:21.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:21.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.793+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:21.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.797+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:21.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.797+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:21.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.797+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:21.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.797+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:21.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.797+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338071d10 2021-03-09T10:23:21.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.797+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8484 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f8344004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:21.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.801+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f8338071d10 2021-03-09T10:23:21.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.801+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033030 con 0x7f8338071d10 2021-03-09T10:23:21.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:21.801+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404b7a0 con 0x7f8338071d10 2021-03-09T10:23:22.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:23:21 smithi146 bash[13136]: debug 2021-03-09T10:23:21.794+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24536 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:23:26.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:23:26.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:26.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8484 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f8344004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:26.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8484 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f8344004c60).stop 2021-03-09T10:23:26.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:26.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:26.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:26.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:26.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:26.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:23:26.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:26.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380149d0 con 0x7f8324003400 2021-03-09T10:23:26.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800daa0 con 0x7f8338069320 2021-03-09T10:23:26.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005700 con 0x7f8324005e00 2021-03-09T10:23:26.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:23:26.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:26.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:26.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:26.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:26.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:26.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:26.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.796+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338072f70 con 0x7f8338069320 2021-03-09T10:23:26.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.800+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a500 secure :-1 s=READY pgs=8490 cs=0 l=1 rev1=1 rx=0x7f833c0614c0 tx=0x7f833c00c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:26.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.800+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338069320 2021-03-09T10:23:26.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.800+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d2d0 con 0x7f8338069320 2021-03-09T10:23:26.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:26.804+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c022730 con 0x7f8338069320 2021-03-09T10:23:27.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:23:26 smithi146 bash[13136]: debug 2021-03-09T10:23:26.798+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24560 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:23:31.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:23:31.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:31.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a500 secure :-1 s=READY pgs=8490 cs=0 l=1 rev1=1 rx=0x7f833c0614c0 tx=0x7f833c00c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:31.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a500 secure :-1 s=READY pgs=8490 cs=0 l=1 rev1=1 rx=0x7f833c0614c0 tx=0x7f833c00c2c0).stop 2021-03-09T10:23:31.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800a500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:31.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a500 unknown :-1 s=CLOSED pgs=8490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:31.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:31.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:31.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:31.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005700 con 0x7f8324003400 2021-03-09T10:23:31.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062970 con 0x7f8338071d10 2021-03-09T10:23:31.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023ff0 con 0x7f8324005e00 2021-03-09T10:23:31.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:23:31.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:31.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:31.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:23:31.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.800+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:31.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.804+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:31.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.804+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:31.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.804+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:31.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.804+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:31.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.804+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028830 con 0x7f8338071d10 2021-03-09T10:23:31.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.804+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8499 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:31.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.804+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404b210 con 0x7f8338071d10 2021-03-09T10:23:31.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.804+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033050 con 0x7f8338071d10 2021-03-09T10:23:31.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:31.808+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338071d10 2021-03-09T10:23:32.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:23:31 smithi146 bash[13136]: debug 2021-03-09T10:23:31.802+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24588 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:23:36.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:23:36.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8499 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8499 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344021000).stop 2021-03-09T10:23:36.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:36.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:36.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:23:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005700 con 0x7f8324003400 2021-03-09T10:23:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062150 con 0x7f8338069320 2021-03-09T10:23:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028cb0 con 0x7f8324005e00 2021-03-09T10:23:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:23:36.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:36.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:36.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:36.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:36.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:36.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:36.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338069320 2021-03-09T10:23:36.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a220 secure :-1 s=READY pgs=8507 cs=0 l=1 rev1=1 rx=0x7f8334034050 tx=0x7f833404b2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:36.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340261b0 con 0x7f8338069320 2021-03-09T10:23:36.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.808+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404d5f0 con 0x7f8338069320 2021-03-09T10:23:36.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:36.812+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404eca0 con 0x7f8338069320 2021-03-09T10:23:37.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:23:36 smithi146 bash[13136]: debug 2021-03-09T10:23:36.806+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24613 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:23:41.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:23:41.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:41.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a220 secure :-1 s=READY pgs=8507 cs=0 l=1 rev1=1 rx=0x7f8334034050 tx=0x7f833404b2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:41.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a220 secure :-1 s=READY pgs=8507 cs=0 l=1 rev1=1 rx=0x7f8334034050 tx=0x7f833404b2d0).stop 2021-03-09T10:23:41.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801a220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:41.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a220 unknown :-1 s=CLOSED pgs=8507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:41.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:41.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:41.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:41.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:41.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:23:41.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:41.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:23:41.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:41.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005700 con 0x7f8324003400 2021-03-09T10:23:41.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8338071d10 2021-03-09T10:23:41.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380167c0 con 0x7f8324005e00 2021-03-09T10:23:41.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:41.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:41.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:41.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:41.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f8338071d10 2021-03-09T10:23:41.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.812+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8516 cs=0 l=1 rev1=1 rx=0x7f8334034050 tx=0x7f833404b2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:41.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.820+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833403a1f0 con 0x7f8338071d10 2021-03-09T10:23:41.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.820+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404d5f0 con 0x7f8338071d10 2021-03-09T10:23:41.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:41.820+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404b600 con 0x7f8338071d10 2021-03-09T10:23:42.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:23:41 smithi146 bash[13136]: debug 2021-03-09T10:23:41.810+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24639 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:23:46.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:23:46.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:46.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8516 cs=0 l=1 rev1=1 rx=0x7f8334034050 tx=0x7f833404b2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:46.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8516 cs=0 l=1 rev1=1 rx=0x7f8334034050 tx=0x7f833404b2d0).stop 2021-03-09T10:23:46.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:46.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:46.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338009c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:46.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:46.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:46.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:23:46.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:46.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:23:46.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:46.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073290 con 0x7f8324003400 2021-03-09T10:23:46.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005700 con 0x7f8338069320 2021-03-09T10:23:46.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324005e00 2021-03-09T10:23:46.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.812+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338009c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:46.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.816+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:46.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.816+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:46.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.816+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:46.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.816+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:46.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.816+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380690f0 con 0x7f8338069320 2021-03-09T10:23:46.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.816+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338009c70 secure :-1 s=READY pgs=8525 cs=0 l=1 rev1=1 rx=0x7f833404b1c0 tx=0x7f833404b2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:46.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.816+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334060170 con 0x7f8338069320 2021-03-09T10:23:46.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.820+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833406d1f0 con 0x7f8338069320 2021-03-09T10:23:46.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:46.820+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404eca0 con 0x7f8338069320 2021-03-09T10:23:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:23:46 smithi146 bash[13136]: debug 2021-03-09T10:23:46.814+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24664 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:23:51.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338009c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:23:51.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338009c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:51.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338009c70 secure :-1 s=READY pgs=8525 cs=0 l=1 rev1=1 rx=0x7f833404b1c0 tx=0x7f833404b2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:51.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338009c70 secure :-1 s=READY pgs=8525 cs=0 l=1 rev1=1 rx=0x7f833404b1c0 tx=0x7f833404b2d0).stop 2021-03-09T10:23:51.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338009c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:51.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338009c70 unknown :-1 s=CLOSED pgs=8525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:51.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:51.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:51.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:51.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:23:51.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380265a0 con 0x7f8324003400 2021-03-09T10:23:51.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062970 con 0x7f8338071d10 2021-03-09T10:23:51.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:51.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380167c0 con 0x7f8324005e00 2021-03-09T10:23:51.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:23:51.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:51.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.816+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:51.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.820+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:51.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.820+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:51.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.820+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:51.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.820+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:51.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.820+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028830 con 0x7f8338071d10 2021-03-09T10:23:51.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.820+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8533 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:51.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.820+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c09adc0 con 0x7f8338071d10 2021-03-09T10:23:51.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.820+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338071d10 2021-03-09T10:23:51.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:51.824+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c045960 con 0x7f8338071d10 2021-03-09T10:23:52.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:23:51 smithi146 bash[13136]: debug 2021-03-09T10:23:51.822+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24691 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:23:56.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:23:56.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:56.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8533 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:56.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8533 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c061430).stop 2021-03-09T10:23:56.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:56.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:56.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:56.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:56.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338009c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:56.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061500 con 0x7f8324003400 2021-03-09T10:23:56.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073150 con 0x7f8338069320 2021-03-09T10:23:56.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dae0 con 0x7f8324005e00 2021-03-09T10:23:56.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:23:56.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:56.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338009c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:56.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:23:56.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:56.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:56.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:56.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:56.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:56.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338069320 2021-03-09T10:23:56.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338009c70 secure :-1 s=READY pgs=8542 cs=0 l=1 rev1=1 rx=0x7f833c09ad50 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:56.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.824+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00c200 con 0x7f8338069320 2021-03-09T10:23:56.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.828+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338069320 2021-03-09T10:23:56.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:23:56.828+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c022730 con 0x7f8338069320 2021-03-09T10:23:57.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:23:56 smithi146 bash[13136]: debug 2021-03-09T10:23:56.825+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24716 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:24:01.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.824+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338009c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:24:01.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.824+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338009c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:01.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.824+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338009c70 secure :-1 s=READY pgs=8542 cs=0 l=1 rev1=1 rx=0x7f833c09ad50 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:01.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.824+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338009c70 secure :-1 s=READY pgs=8542 cs=0 l=1 rev1=1 rx=0x7f833c09ad50 tx=0x7f833c061430).stop 2021-03-09T10:24:01.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.824+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338009c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:01.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.824+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338009c70 unknown :-1 s=CLOSED pgs=8542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:01.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.824+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:01.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.824+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:01.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.828+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:24:01.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.828+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:01.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.828+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:01.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.828+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:24:01.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.828+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022230 con 0x7f8338071d10 2021-03-09T10:24:01.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.828+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065a00 con 0x7f8324005e00 2021-03-09T10:24:01.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.828+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:24:01.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.828+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:01.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.832+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:01.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.832+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:01.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.832+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:01.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.832+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:01.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.832+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:01.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.832+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802c330 con 0x7f8338071d10 2021-03-09T10:24:01.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.832+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8555 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:01.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.836+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01b740 con 0x7f8338071d10 2021-03-09T10:24:01.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.836+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0294a0 con 0x7f8338071d10 2021-03-09T10:24:01.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:01.836+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01fb80 con 0x7f8338071d10 2021-03-09T10:24:02.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:24:01 smithi146 bash[13136]: debug 2021-03-09T10:24:01.829+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24743 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:24:06.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:24:06.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:06.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8555 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:06.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8555 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c061430).stop 2021-03-09T10:24:06.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:06.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:06.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:06.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:06.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:06.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:24:06.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801adc0 con 0x7f8338069320 2021-03-09T10:24:06.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380096f0 con 0x7f8324005e00 2021-03-09T10:24:06.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:24:06.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:06.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:24:06.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:06.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:06.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:06.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:06.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:06.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:06.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801cc00 con 0x7f8338069320 2021-03-09T10:24:06.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.832+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 secure :-1 s=READY pgs=8559 cs=0 l=1 rev1=1 rx=0x7f8344056490 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:06.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.836+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:24:06.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.836+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338069320 2021-03-09T10:24:06.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:06.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404b190 con 0x7f8338069320 2021-03-09T10:24:07.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:24:06 smithi146 bash[13136]: debug 2021-03-09T10:24:06.833+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24768 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:24:11.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:24:11.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:11.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 secure :-1 s=READY pgs=8559 cs=0 l=1 rev1=1 rx=0x7f8344056490 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:11.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 secure :-1 s=READY pgs=8559 cs=0 l=1 rev1=1 rx=0x7f8344056490 tx=0x7f8344021000).stop 2021-03-09T10:24:11.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801d1e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:11.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 unknown :-1 s=CLOSED pgs=8559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:11.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:11.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:11.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:24:11.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:11.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:11.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801adc0 con 0x7f8324003400 2021-03-09T10:24:11.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065fd0 con 0x7f8338071d10 2021-03-09T10:24:11.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380096f0 con 0x7f8324005e00 2021-03-09T10:24:11.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:24:11.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:11.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:11.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:11.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:11.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:11.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:11.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338071d10 2021-03-09T10:24:11.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.840+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8569 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f833404b2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:11.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340261b0 con 0x7f8338071d10 2021-03-09T10:24:11.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007c30 con 0x7f8338071d10 2021-03-09T10:24:11.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:11.844+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334071b00 con 0x7f8338071d10 2021-03-09T10:24:12.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:24:11 smithi146 bash[13136]: debug 2021-03-09T10:24:11.837+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24795 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:24:16.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:24:16.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:16.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8569 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f833404b2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:16.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8569 cs=0 l=1 rev1=1 rx=0x7f8334009910 tx=0x7f833404b2d0).stop 2021-03-09T10:24:16.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:16.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:16.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:16.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:16.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:16.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:24:16.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801adc0 con 0x7f8338069320 2021-03-09T10:24:16.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324005e00 2021-03-09T10:24:16.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:24:16.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:16.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:24:16.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:16.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:16.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:16.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:16.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:16.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:16.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.843+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380217a0 con 0x7f8338069320 2021-03-09T10:24:16.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.847+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801da00 secure :-1 s=READY pgs=8578 cs=0 l=1 rev1=1 rx=0x7f8334024d00 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:16.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.847+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334071c30 con 0x7f8338069320 2021-03-09T10:24:16.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.847+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007c30 con 0x7f8338069320 2021-03-09T10:24:16.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:16.851+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401a360 con 0x7f8338069320 2021-03-09T10:24:17.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:24:16 smithi146 bash[13136]: debug 2021-03-09T10:24:16.841+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24820 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:24:21.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:24:21.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:21.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801da00 secure :-1 s=READY pgs=8578 cs=0 l=1 rev1=1 rx=0x7f8334024d00 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:21.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801da00 secure :-1 s=READY pgs=8578 cs=0 l=1 rev1=1 rx=0x7f8334024d00 tx=0x7f83340482a0).stop 2021-03-09T10:24:21.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801da00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:21.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801da00 unknown :-1 s=CLOSED pgs=8578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:21.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:21.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:21.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:21.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061500 con 0x7f8324003400 2021-03-09T10:24:21.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065fd0 con 0x7f8338071d10 2021-03-09T10:24:21.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:24:21.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:24:21.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:21.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:21.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:24:21.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:21.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:21.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:21.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:21.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:21.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.847+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380675b0 con 0x7f8338071d10 2021-03-09T10:24:21.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.851+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8585 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:21.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.851+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c057490 con 0x7f8338071d10 2021-03-09T10:24:21.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.851+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0576c0 con 0x7f8338071d10 2021-03-09T10:24:21.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:21.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338071d10 2021-03-09T10:24:22.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:24:21 smithi146 bash[13136]: debug 2021-03-09T10:24:21.845+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24846 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:24:26.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:24:26.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:26.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8585 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:26.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8585 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c061430).stop 2021-03-09T10:24:26.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:26.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:26.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:26.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:26.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:26.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324003400 2021-03-09T10:24:26.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801adc0 con 0x7f8338069320 2021-03-09T10:24:26.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:24:26.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:24:26.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:26.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:24:26.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:26.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:26.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:26.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:26.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:26.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:26.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338025220 con 0x7f8338069320 2021-03-09T10:24:26.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.851+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002280 secure :-1 s=READY pgs=8594 cs=0 l=1 rev1=1 rx=0x7f833c026520 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:26.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c045b70 con 0x7f8338069320 2021-03-09T10:24:26.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c026c60 con 0x7f8338069320 2021-03-09T10:24:26.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:26.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c062430 con 0x7f8338069320 2021-03-09T10:24:27.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:24:26 smithi146 bash[13136]: debug 2021-03-09T10:24:26.849+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24872 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:24:31.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338002280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:24:31.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338002280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:31.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002280 secure :-1 s=READY pgs=8594 cs=0 l=1 rev1=1 rx=0x7f833c026520 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:31.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002280 secure :-1 s=READY pgs=8594 cs=0 l=1 rev1=1 rx=0x7f833c026520 tx=0x7f833c061430).stop 2021-03-09T10:24:31.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338002280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:31.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002280 unknown :-1 s=CLOSED pgs=8594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:31.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:31.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:31.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:31.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324003400 2021-03-09T10:24:31.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380192f0 con 0x7f8338071d10 2021-03-09T10:24:31.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380096f0 con 0x7f8324005e00 2021-03-09T10:24:31.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:24:31.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:31.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:31.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:31.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:31.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-03-09T10:24:31.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:24:31.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:31.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:31.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.855+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800e2e0 con 0x7f8338071d10 2021-03-09T10:24:31.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.859+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8604 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:31.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f8338071d10 2021-03-09T10:24:31.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T10:24:31.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:31.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344007740 con 0x7f8338071d10 2021-03-09T10:24:32.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:24:31 smithi146 bash[13136]: debug 2021-03-09T10:24:31.853+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24898 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:24:36.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:24:36.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:36.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8604 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:36.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8604 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344021000).stop 2021-03-09T10:24:36.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:36.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:36.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:36.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:36.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:36.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380250e0 con 0x7f8324003400 2021-03-09T10:24:36.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338069320 2021-03-09T10:24:36.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011b70 con 0x7f8324005e00 2021-03-09T10:24:36.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:24:36.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:36.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:24:36.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:36.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.859+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:36.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.863+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:36.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.863+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:36.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.863+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:36.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.863+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:36.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.863+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801cc00 con 0x7f8338069320 2021-03-09T10:24:36.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.863+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002280 secure :-1 s=READY pgs=8612 cs=0 l=1 rev1=1 rx=0x7f833c06ad10 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:36.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0268c0 con 0x7f8338069320 2021-03-09T10:24:36.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0268c0 con 0x7f8338069320 2021-03-09T10:24:36.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:36.871+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f8338069320 2021-03-09T10:24:37.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:24:36 smithi146 bash[13136]: debug 2021-03-09T10:24:36.857+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24924 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:24:41.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338002280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:24:41.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338002280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:41.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002280 secure :-1 s=READY pgs=8612 cs=0 l=1 rev1=1 rx=0x7f833c06ad10 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:41.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002280 secure :-1 s=READY pgs=8612 cs=0 l=1 rev1=1 rx=0x7f833c06ad10 tx=0x7f833c061430).stop 2021-03-09T10:24:41.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338002280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:41.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002280 unknown :-1 s=CLOSED pgs=8612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:41.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:41.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:24:41.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:41.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:41.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:41.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801adc0 con 0x7f8324003400 2021-03-09T10:24:41.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338071d10 2021-03-09T10:24:41.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380030d0 con 0x7f8324005e00 2021-03-09T10:24:41.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:41.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:24:41.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.863+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:41.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:41.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:41.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:41.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:41.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338071d10 2021-03-09T10:24:41.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8620 cs=0 l=1 rev1=1 rx=0x7f833c0641a0 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:41.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.871+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c045960 con 0x7f8338071d10 2021-03-09T10:24:41.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.871+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338071d10 2021-03-09T10:24:41.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:41.875+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055340 con 0x7f8338071d10 2021-03-09T10:24:42.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:24:41 smithi146 bash[13136]: debug 2021-03-09T10:24:41.865+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24950 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:24:46.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:24:46.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.867+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:46.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8620 cs=0 l=1 rev1=1 rx=0x7f833c0641a0 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:46.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.867+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8620 cs=0 l=1 rev1=1 rx=0x7f833c0641a0 tx=0x7f833c061430).stop 2021-03-09T10:24:46.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:46.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.867+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:46.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.867+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:46.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.867+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:46.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.867+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:46.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f9c0 con 0x7f8324003400 2021-03-09T10:24:46.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8338069320 2021-03-09T10:24:46.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.867+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380614e0 con 0x7f8324005e00 2021-03-09T10:24:46.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.867+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:24:46.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.867+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:46.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.867+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:24:46.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.867+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:46.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.871+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:46.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.871+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:46.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.871+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:46.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.871+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:46.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.871+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:46.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.871+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380675b0 con 0x7f8338069320 2021-03-09T10:24:46.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.871+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 secure :-1 s=READY pgs=8629 cs=0 l=1 rev1=1 rx=0x7f833c054ac0 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:46.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.875+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0546d0 con 0x7f8338069320 2021-03-09T10:24:46.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.875+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0544f0 con 0x7f8338069320 2021-03-09T10:24:46.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:46.879+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0552c0 con 0x7f8338069320 2021-03-09T10:24:47.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:24:46 smithi146 bash[13136]: debug 2021-03-09T10:24:46.869+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 24976 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:24:51.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:24:51.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:51.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 secure :-1 s=READY pgs=8629 cs=0 l=1 rev1=1 rx=0x7f833c054ac0 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:51.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 secure :-1 s=READY pgs=8629 cs=0 l=1 rev1=1 rx=0x7f833c054ac0 tx=0x7f833c061430).stop 2021-03-09T10:24:51.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:51.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 unknown :-1 s=CLOSED pgs=8629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:51.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:51.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:51.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:51.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d6a0 con 0x7f8324003400 2021-03-09T10:24:51.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380030d0 con 0x7f8338071d10 2021-03-09T10:24:51.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a300 con 0x7f8324005e00 2021-03-09T10:24:51.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:24:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:24:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:51.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:51.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.875+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801c090 con 0x7f8338071d10 2021-03-09T10:24:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.879+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8639 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:51.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.879+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f8338071d10 2021-03-09T10:24:51.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.879+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T10:24:51.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:51.883+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834403bbb0 con 0x7f8338071d10 2021-03-09T10:24:52.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:24:51 smithi146 bash[13136]: debug 2021-03-09T10:24:51.873+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25001 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:24:56.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:24:56.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:56.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8639 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:56.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8639 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344021000).stop 2021-03-09T10:24:56.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:56.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:56.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:56.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:56.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:56.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067600 con 0x7f8324003400 2021-03-09T10:24:56.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ba00 con 0x7f8338069320 2021-03-09T10:24:56.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024bb0 con 0x7f8324005e00 2021-03-09T10:24:56.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:24:56.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:56.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:24:56.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:56.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.879+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:56.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.883+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:56.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.883+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:56.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.883+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:56.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.883+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:56.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.883+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801cc00 con 0x7f8338069320 2021-03-09T10:24:56.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.883+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 secure :-1 s=READY pgs=8646 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:56.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.883+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01a660 con 0x7f8338069320 2021-03-09T10:24:56.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.883+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01a660 con 0x7f8338069320 2021-03-09T10:24:56.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:24:56.887+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338069320 2021-03-09T10:24:57.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:24:56 smithi146 bash[13136]: debug 2021-03-09T10:24:56.877+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25028 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:25:01.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:25:01.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:01.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 secure :-1 s=READY pgs=8646 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:01.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 secure :-1 s=READY pgs=8646 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c061430).stop 2021-03-09T10:25:01.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:01.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 unknown :-1 s=CLOSED pgs=8646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:01.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:01.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:01.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:01.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380144c0 con 0x7f8324003400 2021-03-09T10:25:01.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338071d10 2021-03-09T10:25:01.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:25:01.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380030d0 con 0x7f8324005e00 2021-03-09T10:25:01.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:01.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:25:01.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:01.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.883+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:01.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.887+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:01.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.887+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:01.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.887+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:01.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.887+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:01.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.887+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338071d10 2021-03-09T10:25:01.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.887+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8656 cs=0 l=1 rev1=1 rx=0x7f8338032b50 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:01.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.891+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400f1b0 con 0x7f8338071d10 2021-03-09T10:25:01.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.891+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T10:25:01.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:01.891+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f8338071d10 2021-03-09T10:25:02.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:25:01 smithi146 bash[13136]: debug 2021-03-09T10:25:01.881+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25053 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:25:06.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:25:06.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:06.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8656 cs=0 l=1 rev1=1 rx=0x7f8338032b50 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:06.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8656 cs=0 l=1 rev1=1 rx=0x7f8338032b50 tx=0x7f8344021000).stop 2021-03-09T10:25:06.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:06.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:06.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:06.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:25:06.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:06.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:06.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:06.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8324003400 2021-03-09T10:25:06.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065a00 con 0x7f8338069320 2021-03-09T10:25:06.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069940 con 0x7f8324005e00 2021-03-09T10:25:06.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:25:06.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:06.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.886+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:06.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:06.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:06.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:06.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:06.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800d450 con 0x7f8338069320 2021-03-09T10:25:06.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 secure :-1 s=READY pgs=8664 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:06.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.894+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f8338069320 2021-03-09T10:25:06.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.894+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c026790 con 0x7f8338069320 2021-03-09T10:25:06.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:06.894+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054d00 con 0x7f8338069320 2021-03-09T10:25:07.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:25:06 smithi146 bash[13136]: debug 2021-03-09T10:25:06.888+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25080 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:25:11.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:25:11.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.890+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:11.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 secure :-1 s=READY pgs=8664 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:11.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.890+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 secure :-1 s=READY pgs=8664 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c061430).stop 2021-03-09T10:25:11.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:11.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.890+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 unknown :-1 s=CLOSED pgs=8664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:11.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.890+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:11.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.890+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:11.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.890+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:11.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324003400 2021-03-09T10:25:11.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380030d0 con 0x7f8338071d10 2021-03-09T10:25:11.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011b70 con 0x7f8324005e00 2021-03-09T10:25:11.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.890+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:25:11.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.890+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:11.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.894+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:11.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.894+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:25:11.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.894+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:11.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.894+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:11.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.894+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:11.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.894+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:11.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.894+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:11.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.894+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068440 con 0x7f8338071d10 2021-03-09T10:25:11.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.894+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8672 cs=0 l=1 rev1=1 rx=0x7f833403b040 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:11.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.898+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334031130 con 0x7f8338071d10 2021-03-09T10:25:11.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.898+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334031130 con 0x7f8338071d10 2021-03-09T10:25:11.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:11.902+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404ed50 con 0x7f8338071d10 2021-03-09T10:25:12.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:25:11 smithi146 bash[13136]: debug 2021-03-09T10:25:11.892+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25105 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:25:16.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:25:16.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:16.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8672 cs=0 l=1 rev1=1 rx=0x7f833403b040 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:16.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8672 cs=0 l=1 rev1=1 rx=0x7f833403b040 tx=0x7f83340482a0).stop 2021-03-09T10:25:16.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:16.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:16.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:16.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:16.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:16.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324003400 2021-03-09T10:25:16.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8338069320 2021-03-09T10:25:16.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ba00 con 0x7f8324005e00 2021-03-09T10:25:16.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:25:16.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:16.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:25:16.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:16.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.898+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:16.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.902+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:16.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.902+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:16.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.902+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:16.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.902+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:16.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.902+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f8338069320 2021-03-09T10:25:16.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.902+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 secure :-1 s=READY pgs=8683 cs=0 l=1 rev1=1 rx=0x7f833402dcb0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:16.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.902+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334033ee0 con 0x7f8338069320 2021-03-09T10:25:16.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.906+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334033ee0 con 0x7f8338069320 2021-03-09T10:25:16.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:16.910+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334016a70 con 0x7f8338069320 2021-03-09T10:25:17.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:25:16 smithi146 bash[13136]: debug 2021-03-09T10:25:16.896+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25132 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:25:21.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:25:21.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:21.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 secure :-1 s=READY pgs=8683 cs=0 l=1 rev1=1 rx=0x7f833402dcb0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:21.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 secure :-1 s=READY pgs=8683 cs=0 l=1 rev1=1 rx=0x7f833402dcb0 tx=0x7f83340482a0).stop 2021-03-09T10:25:21.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:21.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017420 unknown :-1 s=CLOSED pgs=8683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:21.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:21.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:21.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:21.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d3e0 con 0x7f8324003400 2021-03-09T10:25:21.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024bb0 con 0x7f8338071d10 2021-03-09T10:25:21.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338075940 con 0x7f8324005e00 2021-03-09T10:25:21.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:25:21.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:21.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:25:21.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:21.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:21.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:21.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:21.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:21.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:21.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.902+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801c090 con 0x7f8338071d10 2021-03-09T10:25:21.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.906+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8689 cs=0 l=1 rev1=1 rx=0x7f833804da30 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:21.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.906+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T10:25:21.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.906+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T10:25:21.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:21.910+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404ad90 con 0x7f8338071d10 2021-03-09T10:25:22.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:25:21 smithi146 bash[13136]: debug 2021-03-09T10:25:21.900+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25157 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:25:26.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.906+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:25:26.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.906+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:26.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.906+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8689 cs=0 l=1 rev1=1 rx=0x7f833804da30 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:26.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.906+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8689 cs=0 l=1 rev1=1 rx=0x7f833804da30 tx=0x7f8344021000).stop 2021-03-09T10:25:26.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.906+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:26.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.906+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:26.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.906+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:26.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.906+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:26.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.906+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:26.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.906+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380756f0 con 0x7f8324003400 2021-03-09T10:25:26.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.906+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8338069320 2021-03-09T10:25:26.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.906+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d3e0 con 0x7f8324005e00 2021-03-09T10:25:26.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.906+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:25:26.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.906+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:26.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.910+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:25:26.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.910+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:26.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.910+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:26.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.910+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:26.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.910+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:26.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.910+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:26.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.910+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:26.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.910+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801cc00 con 0x7f8338069320 2021-03-09T10:25:26.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.910+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017b30 secure :-1 s=READY pgs=8702 cs=0 l=1 rev1=1 rx=0x7f833c079660 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:26.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.910+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0792c0 con 0x7f8338069320 2021-03-09T10:25:26.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.914+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338069320 2021-03-09T10:25:26.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:26.914+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0792c0 con 0x7f8338069320 2021-03-09T10:25:27.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:25:26 smithi146 bash[13136]: debug 2021-03-09T10:25:26.908+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25184 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:25:31.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.910+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:25:31.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.910+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:31.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.910+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017b30 secure :-1 s=READY pgs=8702 cs=0 l=1 rev1=1 rx=0x7f833c079660 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:31.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.910+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017b30 secure :-1 s=READY pgs=8702 cs=0 l=1 rev1=1 rx=0x7f833c079660 tx=0x7f833c061430).stop 2021-03-09T10:25:31.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.910+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:31.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.910+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017b30 unknown :-1 s=CLOSED pgs=8702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:31.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.910+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:31.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.910+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:31.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.910+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:31.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.910+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324003400 2021-03-09T10:25:31.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.910+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380756f0 con 0x7f8338071d10 2021-03-09T10:25:31.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.910+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8324005e00 2021-03-09T10:25:31.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.910+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:25:31.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.914+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:31.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.914+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:25:31.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.914+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:31.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.914+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:31.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.914+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:31.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.914+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:31.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.914+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:31.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.914+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:31.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.914+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f8338071d10 2021-03-09T10:25:31.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.914+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8707 cs=0 l=1 rev1=1 rx=0x7f833806bcb0 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:31.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.918+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T10:25:31.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.918+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344066e80 con 0x7f8338071d10 2021-03-09T10:25:31.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:31.918+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405eb20 con 0x7f8338071d10 2021-03-09T10:25:32.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:25:31 smithi146 bash[13136]: debug 2021-03-09T10:25:31.912+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25209 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:25:36.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.914+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:25:37.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.914+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:37.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.914+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8707 cs=0 l=1 rev1=1 rx=0x7f833806bcb0 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:37.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.914+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8707 cs=0 l=1 rev1=1 rx=0x7f833806bcb0 tx=0x7f8344021000).stop 2021-03-09T10:25:37.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.914+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:37.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.914+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:37.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.914+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:37.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.914+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:37.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.914+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338024e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:37.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.914+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069940 con 0x7f8324003400 2021-03-09T10:25:37.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.914+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020000 con 0x7f8338069320 2021-03-09T10:25:37.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.914+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024390 con 0x7f8324005e00 2021-03-09T10:25:37.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.918+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:25:37.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.918+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:37.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.918+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338024e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:37.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.918+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:25:37.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.918+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:37.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.918+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:37.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.918+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:37.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.918+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:37.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.918+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:37.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.918+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338069320 2021-03-09T10:25:37.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.918+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338024e40 secure :-1 s=READY pgs=8715 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:37.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.922+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f8338069320 2021-03-09T10:25:37.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.922+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f8338069320 2021-03-09T10:25:37.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:36.926+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c026380 con 0x7f8338069320 2021-03-09T10:25:37.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:25:36 smithi146 bash[13136]: debug 2021-03-09T10:25:36.916+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25236 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:25:41.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338024e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:25:41.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338024e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:41.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338024e40 secure :-1 s=READY pgs=8715 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:41.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338024e40 secure :-1 s=READY pgs=8715 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c061430).stop 2021-03-09T10:25:41.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338024e40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:41.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338024e40 unknown :-1 s=CLOSED pgs=8715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:41.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:41.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:41.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:41.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:25:41.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8338071d10 2021-03-09T10:25:41.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:25:41.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:25:41.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:41.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073e10 con 0x7f8324005e00 2021-03-09T10:25:41.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:41.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:41.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:41.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:41.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:41.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:41.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806a380 con 0x7f8338071d10 2021-03-09T10:25:41.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.922+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8724 cs=0 l=1 rev1=1 rx=0x7f8334047800 tx=0x7f833403b3c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:41.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.926+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404ead0 con 0x7f8338071d10 2021-03-09T10:25:41.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.926+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404b1a0 con 0x7f8338071d10 2021-03-09T10:25:41.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:41.930+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334003140 con 0x7f8338071d10 2021-03-09T10:25:42.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:25:41 smithi146 bash[13136]: debug 2021-03-09T10:25:41.920+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25261 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:25:46.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:25:46.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:46.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8724 cs=0 l=1 rev1=1 rx=0x7f8334047800 tx=0x7f833403b3c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:46.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8724 cs=0 l=1 rev1=1 rx=0x7f8334047800 tx=0x7f833403b3c0).stop 2021-03-09T10:25:46.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:46.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:46.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:46.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:46.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:46.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324003400 2021-03-09T10:25:46.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d3e0 con 0x7f8338069320 2021-03-09T10:25:46.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324005e00 2021-03-09T10:25:46.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:25:46.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:25:46.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:46.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:46.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:46.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:46.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:46.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:46.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:46.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028460 con 0x7f8338069320 2021-03-09T10:25:46.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.926+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017b30 secure :-1 s=READY pgs=8733 cs=0 l=1 rev1=1 rx=0x7f8338014630 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:46.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.930+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01c6c0 con 0x7f8338069320 2021-03-09T10:25:46.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.930+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c6c0 con 0x7f8338069320 2021-03-09T10:25:46.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:46.934+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c026380 con 0x7f8338069320 2021-03-09T10:25:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:25:46 smithi146 bash[13136]: debug 2021-03-09T10:25:46.924+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25288 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:25:51.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:25:51.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:51.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017b30 secure :-1 s=READY pgs=8733 cs=0 l=1 rev1=1 rx=0x7f8338014630 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:51.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017b30 secure :-1 s=READY pgs=8733 cs=0 l=1 rev1=1 rx=0x7f8338014630 tx=0x7f833c061430).stop 2021-03-09T10:25:51.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338017b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:51.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338017b30 unknown :-1 s=CLOSED pgs=8733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:51.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:51.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:51.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:51.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073e10 con 0x7f8324003400 2021-03-09T10:25:51.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8338071d10 2021-03-09T10:25:51.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:25:51.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:25:51.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:51.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:25:51.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:51.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.930+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:51.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.934+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:51.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.934+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:51.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.934+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:51.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.934+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:51.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.934+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338072610 con 0x7f8338071d10 2021-03-09T10:25:51.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.934+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8741 cs=0 l=1 rev1=1 rx=0x7f834401ce20 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:51.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.934+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404bbb0 con 0x7f8338071d10 2021-03-09T10:25:51.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.938+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033050 con 0x7f8338071d10 2021-03-09T10:25:51.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:51.938+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404c8f0 con 0x7f8338071d10 2021-03-09T10:25:52.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:25:51 smithi146 bash[13136]: debug 2021-03-09T10:25:51.928+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25313 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:25:56.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:25:56.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:56.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8741 cs=0 l=1 rev1=1 rx=0x7f834401ce20 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:56.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8741 cs=0 l=1 rev1=1 rx=0x7f834401ce20 tx=0x7f8344021000).stop 2021-03-09T10:25:56.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:56.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:56.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:56.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:56.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:56.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f4b0 con 0x7f8324003400 2021-03-09T10:25:56.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020000 con 0x7f8338069320 2021-03-09T10:25:56.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324005e00 2021-03-09T10:25:56.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:25:56.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:56.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:56.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:25:56.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.933+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:56.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.937+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:56.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.937+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:56.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.937+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:56.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.937+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:56.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.937+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e840 con 0x7f8338069320 2021-03-09T10:25:56.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.937+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020190 secure :-1 s=READY pgs=8750 cs=0 l=1 rev1=1 rx=0x7f833c0797d0 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:56.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.937+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c054960 con 0x7f8338069320 2021-03-09T10:25:56.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.937+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c054b90 con 0x7f8338069320 2021-03-09T10:25:56.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:25:56.941+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01c590 con 0x7f8338069320 2021-03-09T10:25:57.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:25:56 smithi146 bash[13136]: debug 2021-03-09T10:25:56.936+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25340 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:26:01.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.937+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338020190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:26:01.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.937+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338020190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:01.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.937+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020190 secure :-1 s=READY pgs=8750 cs=0 l=1 rev1=1 rx=0x7f833c0797d0 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:01.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.937+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020190 secure :-1 s=READY pgs=8750 cs=0 l=1 rev1=1 rx=0x7f833c0797d0 tx=0x7f833c061430).stop 2021-03-09T10:26:01.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.937+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338020190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:01.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.937+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020190 unknown :-1 s=CLOSED pgs=8750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:01.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.937+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:01.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.937+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:01.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.937+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:01.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.937+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072a20 con 0x7f8324003400 2021-03-09T10:26:01.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.937+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8338071d10 2021-03-09T10:26:01.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.937+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:26:01.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.937+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:26:01.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.937+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:01.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.941+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:26:01.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.941+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:01.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.941+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:01.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.945+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:01.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.945+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:01.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.945+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:01.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.945+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:01.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.945+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e840 con 0x7f8338071d10 2021-03-09T10:26:01.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.945+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8761 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:01.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.945+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05b620 con 0x7f8338071d10 2021-03-09T10:26:01.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.945+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c045960 con 0x7f8338071d10 2021-03-09T10:26:01.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:01.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c036c20 con 0x7f8338071d10 2021-03-09T10:26:02.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:26:01 smithi146 bash[13136]: debug 2021-03-09T10:26:01.940+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25365 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:26:06.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.941+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:26:06.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.941+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:06.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.941+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8761 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:06.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.941+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8761 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c061430).stop 2021-03-09T10:26:06.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.941+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:06.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.941+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:06.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.941+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338018340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:06.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.941+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:06.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.941+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:06.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.941+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073e10 con 0x7f8324003400 2021-03-09T10:26:06.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.941+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002ea0 con 0x7f8338069320 2021-03-09T10:26:06.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.941+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a300 con 0x7f8324005e00 2021-03-09T10:26:06.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.941+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:26:06.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.941+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:06.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.941+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:26:06.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.941+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:06.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.945+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338018340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:06.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.945+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:06.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.945+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:06.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.945+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:06.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.945+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:06.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.945+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380744d0 con 0x7f8338069320 2021-03-09T10:26:06.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.945+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338018340 secure :-1 s=READY pgs=8767 cs=0 l=1 rev1=1 rx=0x7f833404f9b0 tx=0x7f833403b3c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:06.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404fd50 con 0x7f8338069320 2021-03-09T10:26:06.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334031130 con 0x7f8338069320 2021-03-09T10:26:06.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:06.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404eca0 con 0x7f8338069320 2021-03-09T10:26:07.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:26:06 smithi146 bash[13136]: debug 2021-03-09T10:26:06.948+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25392 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:26:11.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.945+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338018340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:26:11.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.945+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338018340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:11.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.945+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338018340 secure :-1 s=READY pgs=8767 cs=0 l=1 rev1=1 rx=0x7f833404f9b0 tx=0x7f833403b3c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:11.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.945+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338018340 secure :-1 s=READY pgs=8767 cs=0 l=1 rev1=1 rx=0x7f833404f9b0 tx=0x7f833403b3c0).stop 2021-03-09T10:26:11.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338018340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338018340 unknown :-1 s=CLOSED pgs=8767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:11.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:26:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010610 con 0x7f8324003400 2021-03-09T10:26:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073600 con 0x7f8338071d10 2021-03-09T10:26:11.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324005e00 2021-03-09T10:26:11.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:11.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:26:11.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:11.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:11.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:11.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:11.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:11.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380666a0 con 0x7f8338071d10 2021-03-09T10:26:11.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8776 cs=0 l=1 rev1=1 rx=0x7f834400f780 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:11.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f8338071d10 2021-03-09T10:26:11.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.953+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T10:26:11.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:11.957+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400f1b0 con 0x7f8338071d10 2021-03-09T10:26:12.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:26:11 smithi146 bash[13136]: debug 2021-03-09T10:26:11.952+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25417 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:26:16.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.949+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:26:16.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.949+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:16.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.949+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8776 cs=0 l=1 rev1=1 rx=0x7f834400f780 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:16.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.949+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8776 cs=0 l=1 rev1=1 rx=0x7f834400f780 tx=0x7f8344021000).stop 2021-03-09T10:26:16.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:16.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.949+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:16.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.949+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:16.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.949+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:16.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.949+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:16.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e250 con 0x7f8324003400 2021-03-09T10:26:16.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a300 con 0x7f8338069320 2021-03-09T10:26:16.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8324005e00 2021-03-09T10:26:16.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.949+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:26:16.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.949+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:26:16.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.949+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:16.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.949+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:16.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.953+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:16.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.953+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:16.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.953+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:16.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.953+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:16.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.953+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:16.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.953+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e840 con 0x7f8338069320 2021-03-09T10:26:16.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.953+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 secure :-1 s=READY pgs=8785 cs=0 l=1 rev1=1 rx=0x7f833403b390 tx=0x7f83340283d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:16.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.957+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404ead0 con 0x7f8338069320 2021-03-09T10:26:16.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.957+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404f2b0 con 0x7f8338069320 2021-03-09T10:26:16.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:16.957+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404dde0 con 0x7f8338069320 2021-03-09T10:26:17.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:26:16 smithi146 bash[13136]: debug 2021-03-09T10:26:16.955+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25443 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:26:21.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:26:21.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:21.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 secure :-1 s=READY pgs=8785 cs=0 l=1 rev1=1 rx=0x7f833403b390 tx=0x7f83340283d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:21.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 secure :-1 s=READY pgs=8785 cs=0 l=1 rev1=1 rx=0x7f833403b390 tx=0x7f83340283d0).stop 2021-03-09T10:26:21.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:21.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338025e80 unknown :-1 s=CLOSED pgs=8785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:21.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:21.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:21.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:21.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324003400 2021-03-09T10:26:21.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073600 con 0x7f8338071d10 2021-03-09T10:26:21.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073840 con 0x7f8324005e00 2021-03-09T10:26:21.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:26:21.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:21.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:26:21.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:21.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:21.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:21.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:21.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:21.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:21.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338073210 con 0x7f8338071d10 2021-03-09T10:26:21.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.957+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8793 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:21.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.961+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338071d10 2021-03-09T10:26:21.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.961+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c029d30 con 0x7f8338071d10 2021-03-09T10:26:21.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:21.965+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c045960 con 0x7f8338071d10 2021-03-09T10:26:22.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:26:21 smithi146 bash[13136]: debug 2021-03-09T10:26:21.959+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25470 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:26:26.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:26:26.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:26.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8793 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:26.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8793 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c061430).stop 2021-03-09T10:26:26.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:26.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:26.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:26.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:26.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:26.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073840 con 0x7f8324003400 2021-03-09T10:26:26.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8338069320 2021-03-09T10:26:26.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380237c0 con 0x7f8324005e00 2021-03-09T10:26:26.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:26:26.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:26.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:26:26.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:26.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.961+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:26.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.965+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:26.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.965+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:26.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.965+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:26.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.965+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:26.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.965+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806dc90 con 0x7f8338069320 2021-03-09T10:26:26.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.973+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801fe20 secure :-1 s=READY pgs=8807 cs=0 l=1 rev1=1 rx=0x7f83380157c0 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:26.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c07a520 con 0x7f8338069320 2021-03-09T10:26:26.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c8f0 con 0x7f8338069320 2021-03-09T10:26:26.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:26.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05c380 con 0x7f8338069320 2021-03-09T10:26:27.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:26:26 smithi146 bash[13136]: debug 2021-03-09T10:26:26.963+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25495 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:26:31.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:26:31.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:31.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801fe20 secure :-1 s=READY pgs=8807 cs=0 l=1 rev1=1 rx=0x7f83380157c0 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:31.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801fe20 secure :-1 s=READY pgs=8807 cs=0 l=1 rev1=1 rx=0x7f83380157c0 tx=0x7f833c061430).stop 2021-03-09T10:26:31.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801fe20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:31.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801fe20 unknown :-1 s=CLOSED pgs=8807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:31.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:31.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:31.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338060f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:31.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324003400 2021-03-09T10:26:31.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065490 con 0x7f8338071d10 2021-03-09T10:26:31.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f4b0 con 0x7f8324005e00 2021-03-09T10:26:31.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:26:31.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:31.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:26:31.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:31.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338060f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:31.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:31.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:31.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:31.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.965+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:31.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.969+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e840 con 0x7f8338071d10 2021-03-09T10:26:31.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.969+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338060f40 secure :-1 s=READY pgs=8812 cs=0 l=1 rev1=1 rx=0x7f833c026040 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:31.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.969+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00fe70 con 0x7f8338071d10 2021-03-09T10:26:31.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.969+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00fe70 con 0x7f8338071d10 2021-03-09T10:26:31.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:31.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f8338071d10 2021-03-09T10:26:32.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:26:31 smithi146 bash[13136]: debug 2021-03-09T10:26:31.967+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25522 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:26:36.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338060f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:26:36.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338060f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:36.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338060f40 secure :-1 s=READY pgs=8812 cs=0 l=1 rev1=1 rx=0x7f833c026040 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:36.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338060f40 secure :-1 s=READY pgs=8812 cs=0 l=1 rev1=1 rx=0x7f833c026040 tx=0x7f833c061430).stop 2021-03-09T10:26:36.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338060f40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:36.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338060f40 unknown :-1 s=CLOSED pgs=8812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:36.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:36.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:36.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:36.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cf20 con 0x7f8324003400 2021-03-09T10:26:36.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073840 con 0x7f8338069320 2021-03-09T10:26:36.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f9c0 con 0x7f8324005e00 2021-03-09T10:26:36.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:26:36.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:36.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:26:36.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:36.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:36.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:36.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:36.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:36.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:36.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338021c70 con 0x7f8338069320 2021-03-09T10:26:36.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020440 secure :-1 s=READY pgs=8819 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334071f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:36.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8338069320 2021-03-09T10:26:36.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334063180 con 0x7f8338069320 2021-03-09T10:26:36.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:36.981+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404eca0 con 0x7f8338069320 2021-03-09T10:26:37.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:26:36 smithi146 bash[13136]: debug 2021-03-09T10:26:36.975+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25547 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:26:41.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.973+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338020440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:26:41.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.973+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338020440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:41.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.973+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020440 secure :-1 s=READY pgs=8819 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334071f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:41.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.973+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020440 secure :-1 s=READY pgs=8819 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334071f50).stop 2021-03-09T10:26:41.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338020440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:41.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338020440 unknown :-1 s=CLOSED pgs=8819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:41.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:41.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:41.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.973+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:41.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380237c0 con 0x7f8324003400 2021-03-09T10:26:41.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8338071d10 2021-03-09T10:26:41.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.973+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065490 con 0x7f8324005e00 2021-03-09T10:26:41.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.973+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:26:41.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.973+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:26:41.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.973+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:41.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.973+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:41.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.977+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:41.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.977+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:41.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.977+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:41.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.977+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:41.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.977+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:41.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.977+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b6c0 con 0x7f8338071d10 2021-03-09T10:26:41.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.981+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8832 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:41.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.981+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c007a60 con 0x7f8338071d10 2021-03-09T10:26:41.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.981+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c007650 con 0x7f8338071d10 2021-03-09T10:26:41.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:41.985+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338071d10 2021-03-09T10:26:42.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:26:41 smithi146 bash[13136]: debug 2021-03-09T10:26:41.979+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25574 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:26:46.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.976+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:26:46.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.976+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:46.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.976+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8832 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:46.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.976+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8832 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c061430).stop 2021-03-09T10:26:46.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.976+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:46.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.976+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:46.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.976+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:46.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.976+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:46.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.976+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:46.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.976+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324003400 2021-03-09T10:26:46.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.976+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069b40 con 0x7f8338069320 2021-03-09T10:26:46.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.976+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073840 con 0x7f8324005e00 2021-03-09T10:26:46.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.976+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:26:46.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.976+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:46.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.980+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:46.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.980+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:26:46.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.980+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:46.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.980+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:46.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.980+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:46.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.980+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:46.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.980+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:46.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.980+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338067570 con 0x7f8338069320 2021-03-09T10:26:46.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.980+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 secure :-1 s=READY pgs=8837 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:46.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.980+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404af00 con 0x7f8338069320 2021-03-09T10:26:46.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.984+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338069320 2021-03-09T10:26:46.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:46.988+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400f1b0 con 0x7f8338069320 2021-03-09T10:26:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:26:46 smithi146 bash[13136]: debug 2021-03-09T10:26:46.983+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25598 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:26:51.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:26:51.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:51.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 secure :-1 s=READY pgs=8837 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:51.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 secure :-1 s=READY pgs=8837 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344021000).stop 2021-03-09T10:26:51.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338019650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:51.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338019650 unknown :-1 s=CLOSED pgs=8837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:51.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:51.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:51.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:51.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338071b30 con 0x7f8324003400 2021-03-09T10:26:51.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380214c0 con 0x7f8338071d10 2021-03-09T10:26:51.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015180 con 0x7f8324005e00 2021-03-09T10:26:51.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:26:51.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:51.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:26:51.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:51.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:51.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:51.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:51.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:51.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:51.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380105c0 con 0x7f8338071d10 2021-03-09T10:26:51.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.984+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801a360 secure :-1 s=READY pgs=8845 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:51.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.988+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344010bb0 con 0x7f8338071d10 2021-03-09T10:26:51.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.988+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404aa60 con 0x7f8338071d10 2021-03-09T10:26:51.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:51.992+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344010bb0 con 0x7f8338071d10 2021-03-09T10:26:52.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:26:51 smithi146 bash[13136]: debug 2021-03-09T10:26:51.987+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25626 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:26:56.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:26:56.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:56.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801a360 secure :-1 s=READY pgs=8845 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:56.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801a360 secure :-1 s=READY pgs=8845 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344021000).stop 2021-03-09T10:26:56.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833801a360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:56.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833801a360 unknown :-1 s=CLOSED pgs=8845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:56.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380720f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:56.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:56.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:56.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d160 con 0x7f8324003400 2021-03-09T10:26:56.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380237c0 con 0x7f8338069320 2021-03-09T10:26:56.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021cc0 con 0x7f8324005e00 2021-03-09T10:26:56.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:26:56.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:26:56.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:56.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:56.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:56.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:56.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:56.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:56.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:56.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800abf0 con 0x7f8338069320 2021-03-09T10:26:56.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.988+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380720f0 secure :-1 s=READY pgs=8854 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334071f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:56.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.992+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8338069320 2021-03-09T10:26:56.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.992+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8338069320 2021-03-09T10:26:57.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:26:56.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404eca0 con 0x7f8338069320 2021-03-09T10:26:57.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:26:56 smithi146 bash[13136]: debug 2021-03-09T10:26:56.991+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25650 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:27:01.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.988+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:27:01.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.988+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:01.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.988+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380720f0 secure :-1 s=READY pgs=8854 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334071f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:01.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.988+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380720f0 secure :-1 s=READY pgs=8854 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334071f50).stop 2021-03-09T10:27:01.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.988+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380720f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:01.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.988+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380720f0 unknown :-1 s=CLOSED pgs=8854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:01.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.988+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:01.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.988+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338013fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:01.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.988+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:01.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.988+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d160 con 0x7f8324003400 2021-03-09T10:27:01.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.988+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008060 con 0x7f8338071d10 2021-03-09T10:27:01.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.988+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:27:01.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.992+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:27:01.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.992+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:01.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.992+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:27:01.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.992+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:01.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.992+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338013fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:01.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.992+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:01.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.992+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:01.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.992+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:01.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.992+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:01.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.992+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338008870 con 0x7f8338071d10 2021-03-09T10:27:01.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.992+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338013fd0 secure :-1 s=READY pgs=8863 cs=0 l=1 rev1=1 rx=0x7f834405df90 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:02.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405ddd0 con 0x7f8338071d10 2021-03-09T10:27:02.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:01.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T10:27:02.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:02.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344014580 con 0x7f8338071d10 2021-03-09T10:27:02.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:27:01 smithi146 bash[13136]: debug 2021-03-09T10:27:01.995+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25678 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:27:06.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338013fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:27:06.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338013fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:06.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338013fd0 secure :-1 s=READY pgs=8863 cs=0 l=1 rev1=1 rx=0x7f834405df90 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:06.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338013fd0 secure :-1 s=READY pgs=8863 cs=0 l=1 rev1=1 rx=0x7f834405df90 tx=0x7f8344021000).stop 2021-03-09T10:27:06.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338013fd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:06.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338013fd0 unknown :-1 s=CLOSED pgs=8863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:06.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:07.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:07.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:07.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:27:07.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:07.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:27:07.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:07.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:07.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e5b0 con 0x7f8324003400 2021-03-09T10:27:07.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c800 con 0x7f8338069320 2021-03-09T10:27:07.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380756f0 con 0x7f8324005e00 2021-03-09T10:27:07.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:07.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:07.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:07.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:06.996+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:07.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:07.000+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801d3e0 con 0x7f8338069320 2021-03-09T10:27:07.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:07.000+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380091b0 secure :-1 s=READY pgs=8871 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:07.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:07.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344054640 con 0x7f8338069320 2021-03-09T10:27:07.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:07.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8338069320 2021-03-09T10:27:07.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:07.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405d9c0 con 0x7f8338069320 2021-03-09T10:27:07.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:27:07 smithi146 bash[13136]: debug 2021-03-09T10:27:06.999+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25702 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:27:12.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:27:12.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:12.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380091b0 secure :-1 s=READY pgs=8871 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:12.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380091b0 secure :-1 s=READY pgs=8871 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344021000).stop 2021-03-09T10:27:12.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380091b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:12.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380091b0 unknown :-1 s=CLOSED pgs=8871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:12.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:12.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:12.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:12.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e5b0 con 0x7f8324003400 2021-03-09T10:27:12.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380284d0 con 0x7f8338071d10 2021-03-09T10:27:12.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dc60 con 0x7f8324005e00 2021-03-09T10:27:12.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:27:12.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:12.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:27:12.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:12.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.000+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:12.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.004+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:12.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.004+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:12.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.004+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:12.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.004+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:12.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.004+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800dc40 con 0x7f8338071d10 2021-03-09T10:27:12.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.004+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8881 cs=0 l=1 rev1=1 rx=0x7f834405ee40 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:12.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8338071d10 2021-03-09T10:27:12.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344054640 con 0x7f8338071d10 2021-03-09T10:27:12.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:12.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338071d10 2021-03-09T10:27:12.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:27:12 smithi146 bash[13136]: debug 2021-03-09T10:27:12.003+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25730 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:27:17.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:27:17.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:17.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8881 cs=0 l=1 rev1=1 rx=0x7f834405ee40 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:17.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8881 cs=0 l=1 rev1=1 rx=0x7f834405ee40 tx=0x7f8344021000).stop 2021-03-09T10:27:17.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:17.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:17.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:17.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:17.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338074970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:17.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:27:17.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:17.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:27:17.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:17.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e5b0 con 0x7f8324003400 2021-03-09T10:27:17.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380729d0 con 0x7f8338069320 2021-03-09T10:27:17.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d3d0 con 0x7f8324005e00 2021-03-09T10:27:17.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338074970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:17.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:17.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:17.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:17.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:17.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e680 con 0x7f8338069320 2021-03-09T10:27:17.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.008+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338074970 secure :-1 s=READY pgs=8890 cs=0 l=1 rev1=1 rx=0x7f833801e390 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:17.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338069320 2021-03-09T10:27:17.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c029ce0 con 0x7f8338069320 2021-03-09T10:27:17.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:17.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f8338069320 2021-03-09T10:27:17.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:27:17 smithi146 bash[13136]: debug 2021-03-09T10:27:17.011+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25754 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:27:22.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338074970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:27:22.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338074970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:22.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338074970 secure :-1 s=READY pgs=8890 cs=0 l=1 rev1=1 rx=0x7f833801e390 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:22.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338074970 secure :-1 s=READY pgs=8890 cs=0 l=1 rev1=1 rx=0x7f833801e390 tx=0x7f833c061430).stop 2021-03-09T10:27:22.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338074970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:22.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338074970 unknown :-1 s=CLOSED pgs=8890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:22.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833800a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:22.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:22.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:22.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:27:22.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:22.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019be0 con 0x7f8324003400 2021-03-09T10:27:22.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066e70 con 0x7f8338071d10 2021-03-09T10:27:22.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024730 con 0x7f8324005e00 2021-03-09T10:27:22.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:27:22.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:22.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833800a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:22.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:22.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:22.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:22.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:22.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.012+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802cb40 con 0x7f8338071d10 2021-03-09T10:27:22.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.016+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833800a0a0 secure :-1 s=READY pgs=8902 cs=0 l=1 rev1=1 rx=0x7f8334057010 tx=0x7f8334071f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:22.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.016+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334070270 con 0x7f8338071d10 2021-03-09T10:27:22.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.020+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334031130 con 0x7f8338071d10 2021-03-09T10:27:22.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:22.020+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334064de0 con 0x7f8338071d10 2021-03-09T10:27:22.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:27:22 smithi146 bash[13136]: debug 2021-03-09T10:27:22.015+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25782 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:27:27.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.012+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833800a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:27:27.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.012+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833800a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:27.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.012+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833800a0a0 secure :-1 s=READY pgs=8902 cs=0 l=1 rev1=1 rx=0x7f8334057010 tx=0x7f8334071f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:27.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.012+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833800a0a0 secure :-1 s=READY pgs=8902 cs=0 l=1 rev1=1 rx=0x7f8334057010 tx=0x7f8334071f50).stop 2021-03-09T10:27:27.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833800a0a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:27.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833800a0a0 unknown :-1 s=CLOSED pgs=8902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:27.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380645b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:27.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:27.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.012+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:27.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f9e0 con 0x7f8324003400 2021-03-09T10:27:27.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010810 con 0x7f8338069320 2021-03-09T10:27:27.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8324005e00 2021-03-09T10:27:27.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.012+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:27:27.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.012+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:27:27.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.012+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:27.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.012+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:27.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.016+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:27.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.016+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:27.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.016+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:27.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.016+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:27.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.016+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:27.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.016+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338021c70 con 0x7f8338069320 2021-03-09T10:27:27.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.016+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380645b0 secure :-1 s=READY pgs=8906 cs=0 l=1 rev1=1 rx=0x7f833403a6f0 tx=0x7f8334070df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:27.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.020+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404eeb0 con 0x7f8338069320 2021-03-09T10:27:27.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.020+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8338069320 2021-03-09T10:27:27.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:27.020+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401a960 con 0x7f8338069320 2021-03-09T10:27:27.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:27:27 smithi146 bash[13136]: debug 2021-03-09T10:27:27.018+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25806 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:27:32.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380645b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:27:32.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380645b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:32.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380645b0 secure :-1 s=READY pgs=8906 cs=0 l=1 rev1=1 rx=0x7f833403a6f0 tx=0x7f8334070df0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:32.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380645b0 secure :-1 s=READY pgs=8906 cs=0 l=1 rev1=1 rx=0x7f833403a6f0 tx=0x7f8334070df0).stop 2021-03-09T10:27:32.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380645b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:32.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380645b0 unknown :-1 s=CLOSED pgs=8906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:32.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:32.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:32.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:32.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800bbf0 con 0x7f8324003400 2021-03-09T10:27:32.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8338071d10 2021-03-09T10:27:32.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024730 con 0x7f8324005e00 2021-03-09T10:27:32.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:27:32.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:32.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:27:32.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:32.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:32.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:32.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:32.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:32.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:32.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338067620 con 0x7f8338071d10 2021-03-09T10:27:32.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.020+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8916 cs=0 l=1 rev1=1 rx=0x7f8338015730 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:32.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.024+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c045a50 con 0x7f8338071d10 2021-03-09T10:27:32.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.024+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c045bb0 con 0x7f8338071d10 2021-03-09T10:27:32.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:32.028+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c310 con 0x7f8338071d10 2021-03-09T10:27:32.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:27:32 smithi146 bash[13136]: debug 2021-03-09T10:27:32.022+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25834 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:27:37.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:27:37.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:37.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8916 cs=0 l=1 rev1=1 rx=0x7f8338015730 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:37.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8916 cs=0 l=1 rev1=1 rx=0x7f8338015730 tx=0x7f833c061430).stop 2021-03-09T10:27:37.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:37.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:37.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338064fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:37.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:37.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:37.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028ca0 con 0x7f8324003400 2021-03-09T10:27:37.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8338069320 2021-03-09T10:27:37.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f9e0 con 0x7f8324005e00 2021-03-09T10:27:37.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:27:37.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:37.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:27:37.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:37.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338064fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:37.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:37.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:37.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:37.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:37.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.027+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338075900 con 0x7f8338069320 2021-03-09T10:27:37.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.031+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338064fc0 secure :-1 s=READY pgs=8924 cs=0 l=1 rev1=1 rx=0x7f833401d040 tx=0x7f8334004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:37.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340087e0 con 0x7f8338069320 2021-03-09T10:27:37.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334047790 con 0x7f8338069320 2021-03-09T10:27:37.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:37.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334070d20 con 0x7f8338069320 2021-03-09T10:27:37.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:27:37 smithi146 bash[13136]: debug 2021-03-09T10:27:37.026+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25858 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:27:42.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338064fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:27:42.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338064fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:42.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338064fc0 secure :-1 s=READY pgs=8924 cs=0 l=1 rev1=1 rx=0x7f833401d040 tx=0x7f8334004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:42.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338064fc0 secure :-1 s=READY pgs=8924 cs=0 l=1 rev1=1 rx=0x7f833401d040 tx=0x7f8334004c60).stop 2021-03-09T10:27:42.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338064fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:42.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338064fc0 unknown :-1 s=CLOSED pgs=8924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:42.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:42.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:42.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:42.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066e70 con 0x7f8324003400 2021-03-09T10:27:42.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338071d10 2021-03-09T10:27:42.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064390 con 0x7f8324005e00 2021-03-09T10:27:42.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:27:42.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:42.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:42.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:27:42.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.031+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:42.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.035+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:42.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.035+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:42.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.035+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:42.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.035+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:42.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.035+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338067620 con 0x7f8338071d10 2021-03-09T10:27:42.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.035+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8932 cs=0 l=1 rev1=1 rx=0x7f833c026c20 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:42.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c026380 con 0x7f8338071d10 2021-03-09T10:27:42.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0264e0 con 0x7f8338071d10 2021-03-09T10:27:42.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:42.043+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c09a310 con 0x7f8338071d10 2021-03-09T10:27:42.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:27:42 smithi146 bash[13136]: debug 2021-03-09T10:27:42.030+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25886 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:27:47.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.035+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:27:47.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.035+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:47.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.035+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8932 cs=0 l=1 rev1=1 rx=0x7f833c026c20 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:47.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.035+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8932 cs=0 l=1 rev1=1 rx=0x7f833c026c20 tx=0x7f833c061430).stop 2021-03-09T10:27:47.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:47.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.035+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:47.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.035+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:47.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.035+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:47.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.035+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:47.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024730 con 0x7f8324003400 2021-03-09T10:27:47.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380754d0 con 0x7f8338069320 2021-03-09T10:27:47.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025230 con 0x7f8324005e00 2021-03-09T10:27:47.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.035+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:27:47.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.035+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:47.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.035+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:27:47.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.035+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:47.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:47.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:47.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:47.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:47.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:47.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801ab00 con 0x7f8338069320 2021-03-09T10:27:47.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a890 secure :-1 s=READY pgs=8942 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:47.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.043+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c079ae0 con 0x7f8338069320 2021-03-09T10:27:47.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.043+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f8338069320 2021-03-09T10:27:47.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:47.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f8338069320 2021-03-09T10:27:47.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:27:47 smithi146 bash[13136]: debug 2021-03-09T10:27:47.038+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25910 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:27:52.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:27:52.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:52.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a890 secure :-1 s=READY pgs=8942 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:52.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a890 secure :-1 s=READY pgs=8942 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c061430).stop 2021-03-09T10:27:52.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800a890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:52.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.039+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800a890 unknown :-1 s=CLOSED pgs=8942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:52.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.039+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:52.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.039+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:52.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.039+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:52.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028810 con 0x7f8324003400 2021-03-09T10:27:52.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f9e0 con 0x7f8338071d10 2021-03-09T10:27:52.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dc60 con 0x7f8324005e00 2021-03-09T10:27:52.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.039+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:27:52.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.039+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:52.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.039+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:52.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.043+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:27:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.043+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.043+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.043+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:52.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.043+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:52.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.043+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:52.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.043+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065d40 con 0x7f8338071d10 2021-03-09T10:27:52.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.043+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8951 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:52.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8338071d10 2021-03-09T10:27:52.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334033e40 con 0x7f8338071d10 2021-03-09T10:27:52.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:52.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403ba40 con 0x7f8338071d10 2021-03-09T10:27:52.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:27:52 smithi146 bash[13136]: debug 2021-03-09T10:27:52.042+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25938 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:27:57.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.043+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:27:57.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.043+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:57.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.043+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8951 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:57.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.043+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8951 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334004c60).stop 2021-03-09T10:27:57.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:57.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:57.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:57.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380050e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:57.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:57.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380754d0 con 0x7f8324003400 2021-03-09T10:27:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024730 con 0x7f8338069320 2021-03-09T10:27:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025230 con 0x7f8324005e00 2021-03-09T10:27:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:27:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:27:57.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:57.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380050e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:57.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:57.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:57.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:57.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:57.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062960 con 0x7f8338069320 2021-03-09T10:27:57.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.047+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380050e0 secure :-1 s=READY pgs=8960 cs=0 l=1 rev1=1 rx=0x7f834402b5c0 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:57.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402b220 con 0x7f8338069320 2021-03-09T10:27:57.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f8338069320 2021-03-09T10:27:57.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:27:57.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f8338069320 2021-03-09T10:27:57.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:27:57 smithi146 bash[13136]: debug 2021-03-09T10:27:57.046+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25962 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:28:02.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380050e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:28:02.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380050e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:02.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380050e0 secure :-1 s=READY pgs=8960 cs=0 l=1 rev1=1 rx=0x7f834402b5c0 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:02.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380050e0 secure :-1 s=READY pgs=8960 cs=0 l=1 rev1=1 rx=0x7f834402b5c0 tx=0x7f8344021000).stop 2021-03-09T10:28:02.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380050e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:02.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380050e0 unknown :-1 s=CLOSED pgs=8960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:02.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:02.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:02.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:02.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:28:02.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:02.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380730b0 con 0x7f8324003400 2021-03-09T10:28:02.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338071d10 2021-03-09T10:28:02.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380153c0 con 0x7f8324005e00 2021-03-09T10:28:02.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:02.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:28:02.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:02.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:02.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.051+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:02.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:02.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:02.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338061290 con 0x7f8338071d10 2021-03-09T10:28:02.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8967 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:02.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c07c070 con 0x7f8338071d10 2021-03-09T10:28:02.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c007aa0 con 0x7f8338071d10 2021-03-09T10:28:02.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:02.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338071d10 2021-03-09T10:28:02.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:28:02 smithi146 bash[13136]: debug 2021-03-09T10:28:02.050+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 25990 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:28:07.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:28:07.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:07.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8967 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:07.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8967 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c061430).stop 2021-03-09T10:28:07.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:07.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:07.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:07.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:07.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:07.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f9e0 con 0x7f8324003400 2021-03-09T10:28:07.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800adf0 con 0x7f8338069320 2021-03-09T10:28:07.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066e70 con 0x7f8324005e00 2021-03-09T10:28:07.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:28:07.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:07.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:28:07.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:07.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:07.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:07.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:07.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:07.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:07.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014350 con 0x7f8338069320 2021-03-09T10:28:07.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800da00 secure :-1 s=READY pgs=8976 cs=0 l=1 rev1=1 rx=0x7f8338061690 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:07.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0553a0 con 0x7f8338069320 2021-03-09T10:28:07.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f8338069320 2021-03-09T10:28:07.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:07.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0624d0 con 0x7f8338069320 2021-03-09T10:28:07.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:28:07 smithi146 bash[13136]: debug 2021-03-09T10:28:07.054+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26014 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:28:12.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:28:12.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:12.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800da00 secure :-1 s=READY pgs=8976 cs=0 l=1 rev1=1 rx=0x7f8338061690 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:12.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800da00 secure :-1 s=READY pgs=8976 cs=0 l=1 rev1=1 rx=0x7f8338061690 tx=0x7f833c061430).stop 2021-03-09T10:28:12.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800da00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:12.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800da00 unknown :-1 s=CLOSED pgs=8976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:12.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:12.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:12.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:12.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011400 con 0x7f8324003400 2021-03-09T10:28:12.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c300 con 0x7f8338071d10 2021-03-09T10:28:12.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064390 con 0x7f8324005e00 2021-03-09T10:28:12.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:28:12.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:12.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:28:12.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:12.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.059+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:12.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.063+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:12.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.063+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:12.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.063+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:12.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.063+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:12.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.063+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380730b0 con 0x7f8338071d10 2021-03-09T10:28:12.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.063+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8985 cs=0 l=1 rev1=1 rx=0x7f833c045a40 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:12.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.067+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338071d10 2021-03-09T10:28:12.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.067+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f8338071d10 2021-03-09T10:28:12.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:12.071+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0648e0 con 0x7f8338071d10 2021-03-09T10:28:12.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:28:12 smithi146 bash[13136]: debug 2021-03-09T10:28:12.062+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26041 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:28:17.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.063+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:28:17.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.063+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:17.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.063+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8985 cs=0 l=1 rev1=1 rx=0x7f833c045a40 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:17.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.063+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=8985 cs=0 l=1 rev1=1 rx=0x7f833c045a40 tx=0x7f833c061430).stop 2021-03-09T10:28:17.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:17.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.063+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=8985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:17.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.063+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:17.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.063+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:17.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.063+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:17.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:28:17.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380754d0 con 0x7f8338069320 2021-03-09T10:28:17.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f9e0 con 0x7f8324005e00 2021-03-09T10:28:17.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.063+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:28:17.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.063+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:17.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.063+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:28:17.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.063+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:17.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.067+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:17.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.067+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:17.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.067+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:17.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.067+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:17.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.067+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:17.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.067+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380192d0 con 0x7f8338069320 2021-03-09T10:28:17.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.067+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 secure :-1 s=READY pgs=8995 cs=0 l=1 rev1=1 rx=0x7f833c0641a0 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:17.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.071+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338069320 2021-03-09T10:28:17.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.075+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0a1b30 con 0x7f8338069320 2021-03-09T10:28:17.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:17.075+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05c380 con 0x7f8338069320 2021-03-09T10:28:17.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:28:17 smithi146 bash[13136]: debug 2021-03-09T10:28:17.066+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26066 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:28:22.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.067+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:28:22.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.067+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:22.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.067+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 secure :-1 s=READY pgs=8995 cs=0 l=1 rev1=1 rx=0x7f833c0641a0 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:22.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.071+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 secure :-1 s=READY pgs=8995 cs=0 l=1 rev1=1 rx=0x7f833c0641a0 tx=0x7f833c061430).stop 2021-03-09T10:28:22.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.071+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800b620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:22.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.071+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800b620 unknown :-1 s=CLOSED pgs=8995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:22.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.071+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:22.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.071+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:22.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.071+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:22.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.071+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324003400 2021-03-09T10:28:22.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.071+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c300 con 0x7f8338071d10 2021-03-09T10:28:22.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.071+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f9e0 con 0x7f8324005e00 2021-03-09T10:28:22.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.071+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:28:22.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.071+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:22.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.071+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:28:22.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.071+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:22.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.071+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:22.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.075+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:22.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.075+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:22.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.075+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:22.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.075+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:22.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.075+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806cd20 con 0x7f8338071d10 2021-03-09T10:28:22.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.075+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9006 cs=0 l=1 rev1=1 rx=0x7f833402dcb0 tx=0x7f8334004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:22.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.075+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338071d10 2021-03-09T10:28:22.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.079+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334023170 con 0x7f8338071d10 2021-03-09T10:28:22.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:22.083+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403ba40 con 0x7f8338071d10 2021-03-09T10:28:22.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:28:22 smithi146 bash[13136]: debug 2021-03-09T10:28:22.070+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26093 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:28:27.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:28:27.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:27.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9006 cs=0 l=1 rev1=1 rx=0x7f833402dcb0 tx=0x7f8334004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:27.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9006 cs=0 l=1 rev1=1 rx=0x7f833402dcb0 tx=0x7f8334004c60).stop 2021-03-09T10:28:27.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:27.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=9006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:27.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:27.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:27.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:27.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066e70 con 0x7f8324003400 2021-03-09T10:28:27.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dc60 con 0x7f8338069320 2021-03-09T10:28:27.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380754d0 con 0x7f8324005e00 2021-03-09T10:28:27.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:28:27.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:27.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:28:27.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:27.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.074+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:27.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.078+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:27.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.078+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:27.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.078+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:27.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.078+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:27.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.078+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e680 con 0x7f8338069320 2021-03-09T10:28:27.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.078+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 secure :-1 s=READY pgs=9012 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f8334004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:27.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.086+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334056530 con 0x7f8338069320 2021-03-09T10:28:27.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.086+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334033ec0 con 0x7f8338069320 2021-03-09T10:28:27.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:27.086+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334056bf0 con 0x7f8338069320 2021-03-09T10:28:27.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:28:27 smithi146 bash[13136]: debug 2021-03-09T10:28:27.074+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26118 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:28:32.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:28:32.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:32.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 secure :-1 s=READY pgs=9012 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f8334004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:32.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 secure :-1 s=READY pgs=9012 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f8334004c60).stop 2021-03-09T10:28:32.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338007b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:32.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 unknown :-1 s=CLOSED pgs=9012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:32.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:32.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:32.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:32.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:28:32.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f9e0 con 0x7f8338071d10 2021-03-09T10:28:32.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024bb0 con 0x7f8324005e00 2021-03-09T10:28:32.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:28:32.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:32.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:28:32.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:32.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.078+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:32.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.082+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:32.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.082+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:32.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.082+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:32.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.082+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:32.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.082+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ca00 con 0x7f8338071d10 2021-03-09T10:28:32.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.082+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9019 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:32.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.086+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338071d10 2021-03-09T10:28:32.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.086+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338071d10 2021-03-09T10:28:32.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:32.086+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f8338071d10 2021-03-09T10:28:32.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:28:32 smithi146 bash[13136]: debug 2021-03-09T10:28:32.078+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26144 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:28:37.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:28:37.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:37.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9019 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:37.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9019 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c061430).stop 2021-03-09T10:28:37.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:37.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=9019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:37.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:37.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:37.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:28:37.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:37.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:37.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064390 con 0x7f8324003400 2021-03-09T10:28:37.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072ec0 con 0x7f8338069320 2021-03-09T10:28:37.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380754d0 con 0x7f8324005e00 2021-03-09T10:28:37.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:37.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:28:37.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.082+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:37.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.086+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:37.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.086+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:37.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.086+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:37.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.086+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:37.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.086+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014350 con 0x7f8338069320 2021-03-09T10:28:37.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.086+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800cbd0 secure :-1 s=READY pgs=9028 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:37.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.086+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404ecd0 con 0x7f8338069320 2021-03-09T10:28:37.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338069320 2021-03-09T10:28:37.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:37.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400f1e0 con 0x7f8338069320 2021-03-09T10:28:37.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:28:37 smithi146 bash[13136]: debug 2021-03-09T10:28:37.086+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26170 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:28:42.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:28:42.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:42.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800cbd0 secure :-1 s=READY pgs=9028 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:42.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800cbd0 secure :-1 s=READY pgs=9028 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344021000).stop 2021-03-09T10:28:42.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800cbd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:42.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800cbd0 unknown :-1 s=CLOSED pgs=9028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:42.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:42.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:42.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:42.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021670 con 0x7f8324003400 2021-03-09T10:28:42.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8338071d10 2021-03-09T10:28:42.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:28:42.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:28:42.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:42.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:28:42.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:42.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:42.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:42.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:42.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:42.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:42.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338008bd0 con 0x7f8338071d10 2021-03-09T10:28:42.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.090+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9036 cs=0 l=1 rev1=1 rx=0x7f8334052680 tx=0x7f8334004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:42.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334031130 con 0x7f8338071d10 2021-03-09T10:28:42.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334031130 con 0x7f8338071d10 2021-03-09T10:28:42.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:42.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338071d10 2021-03-09T10:28:42.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:28:42 smithi146 bash[13136]: debug 2021-03-09T10:28:42.089+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26196 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:28:47.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:28:47.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:47.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9036 cs=0 l=1 rev1=1 rx=0x7f8334052680 tx=0x7f8334004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:47.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9036 cs=0 l=1 rev1=1 rx=0x7f8334052680 tx=0x7f8334004c60).stop 2021-03-09T10:28:47.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:47.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=9036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:47.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:47.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:47.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:28:47.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:47.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d5d0 con 0x7f8324003400 2021-03-09T10:28:47.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380029a0 con 0x7f8338069320 2021-03-09T10:28:47.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c300 con 0x7f8324005e00 2021-03-09T10:28:47.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:28:47.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:47.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:47.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.094+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:47.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.098+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:47.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.098+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:47.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.098+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:47.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.098+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:47.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.098+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028480 con 0x7f8338069320 2021-03-09T10:28:47.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.098+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 secure :-1 s=READY pgs=9049 cs=0 l=1 rev1=1 rx=0x7f833c079a70 tx=0x7f833c061430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:47.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c079870 con 0x7f8338069320 2021-03-09T10:28:47.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338069320 2021-03-09T10:28:47.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:47.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c062350 con 0x7f8338069320 2021-03-09T10:28:47.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:28:47 smithi146 bash[13136]: debug 2021-03-09T10:28:47.093+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26222 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:28:52.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:28:52.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:52.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 secure :-1 s=READY pgs=9049 cs=0 l=1 rev1=1 rx=0x7f833c079a70 tx=0x7f833c061430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:52.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 secure :-1 s=READY pgs=9049 cs=0 l=1 rev1=1 rx=0x7f833c079a70 tx=0x7f833c061430).stop 2021-03-09T10:28:52.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338007b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:52.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 unknown :-1 s=CLOSED pgs=9049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:52.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:52.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:52.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:52.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072ec0 con 0x7f8324003400 2021-03-09T10:28:52.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f9e0 con 0x7f8338071d10 2021-03-09T10:28:52.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:28:52.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:28:52.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:52.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:28:52.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:52.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.098+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:52.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.102+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:52.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.102+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:52.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.102+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:52.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.102+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:52.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.102+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014350 con 0x7f8338071d10 2021-03-09T10:28:52.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.102+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9054 cs=0 l=1 rev1=1 rx=0x7f833c05c7c0 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:52.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338071d10 2021-03-09T10:28:52.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0546b0 con 0x7f8338071d10 2021-03-09T10:28:52.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:52.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c045160 con 0x7f8338071d10 2021-03-09T10:28:52.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:28:52 smithi146 bash[13136]: debug 2021-03-09T10:28:52.097+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26248 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:28:57.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:28:57.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:57.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9054 cs=0 l=1 rev1=1 rx=0x7f833c05c7c0 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:57.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9054 cs=0 l=1 rev1=1 rx=0x7f833c05c7c0 tx=0x7f833c04de90).stop 2021-03-09T10:28:57.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:57.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=9054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:57.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:57.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:57.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:57.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025a70 con 0x7f8324003400 2021-03-09T10:28:57.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064990 con 0x7f8338069320 2021-03-09T10:28:57.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024bb0 con 0x7f8324005e00 2021-03-09T10:28:57.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:28:57.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:57.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:28:57.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:57.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.102+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:57.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.106+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:57.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.106+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:57.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.106+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:57.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.106+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:57.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.106+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028480 con 0x7f8338069320 2021-03-09T10:28:57.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.106+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 secure :-1 s=READY pgs=9062 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:57.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.110+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403ce20 con 0x7f8338069320 2021-03-09T10:28:57.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.110+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405be80 con 0x7f8338069320 2021-03-09T10:28:57.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:28:57.110+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:28:57.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:28:57 smithi146 bash[13136]: debug 2021-03-09T10:28:57.105+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26274 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:29:02.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.106+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:29:02.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.106+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:02.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.106+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 secure :-1 s=READY pgs=9062 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:02.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.106+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 secure :-1 s=READY pgs=9062 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344021000).stop 2021-03-09T10:29:02.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801d1e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:02.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.106+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 unknown :-1 s=CLOSED pgs=9062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:02.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.106+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:02.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.106+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:02.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.106+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:02.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.106+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:29:02.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.106+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:02.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.106+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:29:02.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.110+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:02.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.110+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:02.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.110+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324003400 2021-03-09T10:29:02.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.110+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8338071d10 2021-03-09T10:29:02.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.110+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380736b0 con 0x7f8324005e00 2021-03-09T10:29:02.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.110+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:02.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.110+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:02.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.110+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:02.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.110+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:02.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.110+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801aa60 con 0x7f8338071d10 2021-03-09T10:29:02.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.110+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9072 cs=0 l=1 rev1=1 rx=0x7f8344072530 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:02.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.114+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440727d0 con 0x7f8338071d10 2021-03-09T10:29:02.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.114+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033030 con 0x7f8338071d10 2021-03-09T10:29:02.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:02.118+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ac90 con 0x7f8338071d10 2021-03-09T10:29:02.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:29:02 smithi146 bash[13136]: debug 2021-03-09T10:29:02.109+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26300 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:29:07.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.110+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:29:07.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.110+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:07.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.110+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9072 cs=0 l=1 rev1=1 rx=0x7f8344072530 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:07.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.110+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9072 cs=0 l=1 rev1=1 rx=0x7f8344072530 tx=0x7f8344021000).stop 2021-03-09T10:29:07.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.110+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:07.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.110+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=9072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:07.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.110+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:07.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.110+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:07.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.110+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:07.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.110+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064990 con 0x7f8324003400 2021-03-09T10:29:07.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.110+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e860 con 0x7f8338069320 2021-03-09T10:29:07.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.110+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011400 con 0x7f8324005e00 2021-03-09T10:29:07.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.114+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:29:07.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.114+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:07.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.114+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:29:07.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.114+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:07.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.114+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:07.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.114+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:07.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.114+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:07.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.114+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:07.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.114+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:07.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.114+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338069f90 con 0x7f8338069320 2021-03-09T10:29:07.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.114+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 secure :-1 s=READY pgs=9080 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:07.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.118+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334052520 con 0x7f8338069320 2021-03-09T10:29:07.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.118+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334052520 con 0x7f8338069320 2021-03-09T10:29:07.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:07.118+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334060170 con 0x7f8338069320 2021-03-09T10:29:07.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:29:07 smithi146 bash[13136]: debug 2021-03-09T10:29:07.113+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26326 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:29:12.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:29:12.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:12.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 secure :-1 s=READY pgs=9080 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:12.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 secure :-1 s=READY pgs=9080 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334004c60).stop 2021-03-09T10:29:12.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338007b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:12.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 unknown :-1 s=CLOSED pgs=9080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:12.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:12.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:12.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:12.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072bd0 con 0x7f8324003400 2021-03-09T10:29:12.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e760 con 0x7f8338071d10 2021-03-09T10:29:12.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f9e0 con 0x7f8324005e00 2021-03-09T10:29:12.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:29:12.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:12.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:12.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:29:12.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.118+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:12.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.122+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:12.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.122+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:12.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.122+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:12.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.122+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:12.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.122+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278f0 con 0x7f8338071d10 2021-03-09T10:29:12.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.122+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9092 cs=0 l=1 rev1=1 rx=0x7f8338064c50 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:12.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.122+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05c380 con 0x7f8338071d10 2021-03-09T10:29:12.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.122+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0546b0 con 0x7f8338071d10 2021-03-09T10:29:12.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:12.126+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338071d10 2021-03-09T10:29:12.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:29:12 smithi146 bash[13136]: debug 2021-03-09T10:29:12.117+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26352 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:29:17.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:29:17.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:17.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9092 cs=0 l=1 rev1=1 rx=0x7f8338064c50 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9092 cs=0 l=1 rev1=1 rx=0x7f8338064c50 tx=0x7f833c04de90).stop 2021-03-09T10:29:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=9092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:29:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065d40 con 0x7f8338069320 2021-03-09T10:29:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380259f0 con 0x7f8324005e00 2021-03-09T10:29:17.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:29:17.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:29:17.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:17.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:17.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:17.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:17.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:17.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:17.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:17.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801cc50 con 0x7f8338069320 2021-03-09T10:29:17.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.121+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 secure :-1 s=READY pgs=9097 cs=0 l=1 rev1=1 rx=0x7f833403a700 tx=0x7f8334004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:17.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.125+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334052450 con 0x7f8338069320 2021-03-09T10:29:17.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.125+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338069320 2021-03-09T10:29:17.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:17.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404ed40 con 0x7f8338069320 2021-03-09T10:29:17.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:29:17 smithi146 bash[13136]: debug 2021-03-09T10:29:17.121+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26378 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:29:22.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:29:22.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:22.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 secure :-1 s=READY pgs=9097 cs=0 l=1 rev1=1 rx=0x7f833403a700 tx=0x7f8334004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:22.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 secure :-1 s=READY pgs=9097 cs=0 l=1 rev1=1 rx=0x7f833403a700 tx=0x7f8334004c60).stop 2021-03-09T10:29:22.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338007b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:22.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338007b40 unknown :-1 s=CLOSED pgs=9097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:22.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:22.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:22.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:22.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ff60 con 0x7f8324003400 2021-03-09T10:29:22.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010b60 con 0x7f8338071d10 2021-03-09T10:29:22.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018a80 con 0x7f8324005e00 2021-03-09T10:29:22.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:29:22.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:22.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:29:22.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:22.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.125+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:22.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.129+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:22.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.129+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:22.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.129+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:22.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.129+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:22.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.129+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380125b0 con 0x7f8338071d10 2021-03-09T10:29:22.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.129+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9106 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f8334004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:22.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400b9a0 con 0x7f8338071d10 2021-03-09T10:29:22.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404ed10 con 0x7f8338071d10 2021-03-09T10:29:22.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:22.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403a560 con 0x7f8338071d10 2021-03-09T10:29:22.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:29:22 smithi146 bash[13136]: debug 2021-03-09T10:29:22.125+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26404 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:29:27.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.129+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:29:27.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.129+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:27.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.129+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9106 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f8334004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:27.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.129+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9106 cs=0 l=1 rev1=1 rx=0x7f833406efd0 tx=0x7f8334004c60).stop 2021-03-09T10:29:27.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:27.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.129+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=9106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:27.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.129+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:27.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.129+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:27.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.129+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:27.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064990 con 0x7f8324003400 2021-03-09T10:29:27.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d30 con 0x7f8338069320 2021-03-09T10:29:27.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:29:27.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.129+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:29:27.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.129+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:29:27.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.129+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:27.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.129+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:27.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.133+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:27.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.133+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:27.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.133+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:27.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.133+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:27.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.133+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:27.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.133+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806cd20 con 0x7f8338069320 2021-03-09T10:29:27.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.133+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380100b0 secure :-1 s=READY pgs=9114 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:27.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405f310 con 0x7f8338069320 2021-03-09T10:29:27.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338069320 2021-03-09T10:29:27.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:27.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:29:27.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:29:27 smithi146 bash[13136]: debug 2021-03-09T10:29:27.129+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26430 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:29:32.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:29:32.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:32.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380100b0 secure :-1 s=READY pgs=9114 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:32.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380100b0 secure :-1 s=READY pgs=9114 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344021000).stop 2021-03-09T10:29:32.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380100b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:32.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380100b0 unknown :-1 s=CLOSED pgs=9114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:32.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:32.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:32.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:32.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629b0 con 0x7f8324003400 2021-03-09T10:29:32.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ff60 con 0x7f8338071d10 2021-03-09T10:29:32.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324005e00 2021-03-09T10:29:32.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:29:32.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:29:32.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:32.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:32.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:32.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:32.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:32.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f8338068700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:32.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f8338068700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:32.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806cd20 con 0x7f8338071d10 2021-03-09T10:29:32.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9124 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:32.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T10:29:32.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338071d10 2021-03-09T10:29:32.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:32.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ac90 con 0x7f8338071d10 2021-03-09T10:29:32.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:29:32 smithi146 bash[13136]: debug 2021-03-09T10:29:32.137+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26456 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:29:37.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.137+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:29:37.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.137+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:37.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.137+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9124 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:37.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.137+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=9124 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344021000).stop 2021-03-09T10:29:37.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:37.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.137+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=9124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:37.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.137+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:37.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.141+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f83380685e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:37.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.141+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:37.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c300 con 0x7f8324003400 2021-03-09T10:29:37.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015570 con 0x7f8338069320 2021-03-09T10:29:37.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380180b0 con 0x7f8324005e00 2021-03-09T10:29:37.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.141+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f83380685e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:29:37.145 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.141+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:29:37.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.141+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f83380685e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:37.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.141+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:37.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.141+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:37.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.141+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:37.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.141+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:37.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.141+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f83380685e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:37.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.141+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f83380685e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:37.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.141+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020a80 con 0x7f8338069320 2021-03-09T10:29:37.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.141+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380100b0 secure :-1 s=READY pgs=9132 cs=0 l=1 rev1=1 rx=0x7f833806d580 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:37.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.145+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338069320 2021-03-09T10:29:37.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.145+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338069320 2021-03-09T10:29:37.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:37.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0040a0 con 0x7f8338069320 2021-03-09T10:29:37.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:29:37 smithi146 bash[13136]: debug 2021-03-09T10:29:37.141+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26482 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:29:42.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:29:42.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:42.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380100b0 secure :-1 s=READY pgs=9132 cs=0 l=1 rev1=1 rx=0x7f833806d580 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:42.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380100b0 secure :-1 s=READY pgs=9132 cs=0 l=1 rev1=1 rx=0x7f833806d580 tx=0x7f833c04de90).stop 2021-03-09T10:29:42.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380100b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:42.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380100b0 unknown :-1 s=CLOSED pgs=9132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:42.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:42.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380254a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:42.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f83380685e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:42.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324003400 2021-03-09T10:29:42.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e860 con 0x7f8338071d10 2021-03-09T10:29:42.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015180 con 0x7f8324005e00 2021-03-09T10:29:42.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:29:42.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:42.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f83380685e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:29:42.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f83380685e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:42.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.145+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380254a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:42.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.153+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:42.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.153+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:42.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.153+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f83380685e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:42.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.153+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f83380685e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:42.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.153+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802dc60 con 0x7f8338071d10 2021-03-09T10:29:42.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.157+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380254a0 secure :-1 s=READY pgs=9144 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:42.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338071d10 2021-03-09T10:29:42.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344017640 con 0x7f8338071d10 2021-03-09T10:29:42.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:42.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ac90 con 0x7f8338071d10 2021-03-09T10:29:42.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:29:42 smithi146 bash[13136]: debug 2021-03-09T10:29:42.145+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26508 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:29:47.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f83380254a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:29:47.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f83380254a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:47.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380254a0 secure :-1 s=READY pgs=9144 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:47.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380254a0 secure :-1 s=READY pgs=9144 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344021000).stop 2021-03-09T10:29:47.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f83380254a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:47.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f83380254a0 unknown :-1 s=CLOSED pgs=9144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:47.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f83380685e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:47.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:47.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:47.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064390 con 0x7f8324003400 2021-03-09T10:29:47.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c300 con 0x7f8338069320 2021-03-09T10:29:47.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380180b0 con 0x7f8324005e00 2021-03-09T10:29:47.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f83380685e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:29:47.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f83380685e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:47.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:29:47.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:47.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.149+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:47.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.153+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:47.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.153+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833806c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:47.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.153+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f83380685e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:47.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.153+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f83380685e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:47.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.153+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338074f40 con 0x7f8338069320 2021-03-09T10:29:47.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.153+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 secure :-1 s=READY pgs=9153 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:47.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402b830 con 0x7f8338069320 2021-03-09T10:29:47.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338069320 2021-03-09T10:29:47.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:47.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440171c0 con 0x7f8338069320 2021-03-09T10:29:47.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:29:47 smithi146 bash[13136]: debug 2021-03-09T10:29:47.153+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26534 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:29:50.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:50.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338075a90 msgr2=0x7f8338023900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2021-03-09T10:29:50.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:50.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338075a90 msgr2=0x7f8338023900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:50.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:50.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338075a90 0x7f8338023900 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c06e000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:50.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:50.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338075a90 0x7f8338023900 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c06e000).stop 2021-03-09T10:29:50.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:50.645+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338075a90 msgr2=0x7f8338023900 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T10:29:50.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:50.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338075a90 msgr2=0x7f8338023900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:50.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:50.645+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338075a90 0x7f8338023900 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:50.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:50.645+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:50.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:50.645+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8334023380 con 0x7f8338071d10 2021-03-09T10:29:50.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:50.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:50.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:50.645+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c06e000).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:52.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:29:52.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:52.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 secure :-1 s=READY pgs=9153 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:52.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 secure :-1 s=READY pgs=9153 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344021000).stop 2021-03-09T10:29:52.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801d1e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:52.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801d1e0 unknown :-1 s=CLOSED pgs=9153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:52.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338023900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:52.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:52.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f83380685a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:52.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d170 con 0x7f8324003400 2021-03-09T10:29:52.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d30 con 0x7f8338075a90 2021-03-09T10:29:52.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:29:52.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338023900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:29:52.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338023900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:52.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f83380685a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:29:52.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f83380685a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:52.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:52.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338023900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:52.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338023900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:52.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 msgr2=0x7f833806c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T10:29:52.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833806c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:52.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 msgr2=0x7f83380685a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:52.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324003400 0x7f83380685a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:52.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f832400ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:52.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83240057f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:52.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f832400b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:52.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d170 con 0x7f8324006200 2021-03-09T10:29:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060d30 con 0x7f8324008870 2021-03-09T10:29:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069140 con 0x7f8324005e00 2021-03-09T10:29:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338075a90 0x7f833806c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-03-09T10:29:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f832400b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:29:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f832400b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83240057f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:29:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83240057f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f832400ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f83240057f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f83240057f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f832400b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f832400b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380613d0 con 0x7f8324008870 2021-03-09T10:29:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.157+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f832400ad00 secure :-1 s=READY pgs=9159 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334063180 con 0x7f8324008870 2021-03-09T10:29:52.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340462b0 con 0x7f8324008870 2021-03-09T10:29:52.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:52.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404ecf0 con 0x7f8324008870 2021-03-09T10:29:52.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:29:52 smithi146 bash[13136]: debug 2021-03-09T10:29:52.156+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26560 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:29:57.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f832400ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:29:57.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f832400ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:57.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f832400ad00 secure :-1 s=READY pgs=9159 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:57.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f832400ad00 secure :-1 s=READY pgs=9159 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334038000).stop 2021-03-09T10:29:57.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f832400ad00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:57.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f832400ad00 unknown :-1 s=CLOSED pgs=9159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:57.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:57.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:57.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338068500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:57.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060bf0 con 0x7f8324006200 2021-03-09T10:29:57.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020ad0 con 0x7f8338069320 2021-03-09T10:29:57.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:29:57.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065490 con 0x7f8324005e00 2021-03-09T10:29:57.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:57.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338071450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:29:57.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338071450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:57.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.161+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338068500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:57.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338071450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338071450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:57.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802dc60 con 0x7f8338069320 2021-03-09T10:29:57.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338068500 secure :-1 s=READY pgs=9168 cs=0 l=1 rev1=1 rx=0x7f833c05c740 tx=0x7f8344012fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:57.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338069320 2021-03-09T10:29:57.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334023380 con 0x7f8338069320 2021-03-09T10:29:57.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:29:57.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055200 con 0x7f8338069320 2021-03-09T10:29:57.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:29:57 smithi146 bash[13136]: debug 2021-03-09T10:29:57.160+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26585 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:30:02.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338068500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:30:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338068500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:02.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338068500 secure :-1 s=READY pgs=9168 cs=0 l=1 rev1=1 rx=0x7f833c05c740 tx=0x7f8344012fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:02.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338068500 secure :-1 s=READY pgs=9168 cs=0 l=1 rev1=1 rx=0x7f833c05c740 tx=0x7f8344012fa0).stop 2021-03-09T10:30:02.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338068500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:02.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.165+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338068500 unknown :-1 s=CLOSED pgs=9168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:02.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.165+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:02.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.165+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:02.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.165+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:02.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ff60 con 0x7f8324006200 2021-03-09T10:30:02.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f4b0 con 0x7f8324008870 2021-03-09T10:30:02.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8324005e00 2021-03-09T10:30:02.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.165+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:30:02.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.165+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:02.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.165+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338071450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:30:02.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.165+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338071450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:02.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.169+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:02.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.173+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338071450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:02.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.173+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338071450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:02.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.173+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:02.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.173+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:02.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.173+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801cc50 con 0x7f8324008870 2021-03-09T10:30:02.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.173+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016b00 secure :-1 s=READY pgs=9179 cs=0 l=1 rev1=1 rx=0x7f833c01e010 tx=0x7f8344012fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:02.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.173+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c042380 con 0x7f8324008870 2021-03-09T10:30:02.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.177+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c042380 con 0x7f8324008870 2021-03-09T10:30:02.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:02.177+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05cb50 con 0x7f8324008870 2021-03-09T10:30:02.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:30:02 smithi146 bash[13136]: debug 2021-03-09T10:30:02.164+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26611 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:30:07.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338016b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:30:07.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338016b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:07.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016b00 secure :-1 s=READY pgs=9179 cs=0 l=1 rev1=1 rx=0x7f833c01e010 tx=0x7f8344012fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:07.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016b00 secure :-1 s=READY pgs=9179 cs=0 l=1 rev1=1 rx=0x7f833c01e010 tx=0x7f8344012fa0).stop 2021-03-09T10:30:07.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338016b00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:07.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016b00 unknown :-1 s=CLOSED pgs=9179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:07.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:07.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:07.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:07.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012880 con 0x7f8324006200 2021-03-09T10:30:07.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065490 con 0x7f8338069320 2021-03-09T10:30:07.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013190 con 0x7f8324005e00 2021-03-09T10:30:07.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:30:07.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:07.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:30:07.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:07.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.168+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:07.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.172+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:07.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.172+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:07.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.172+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:07.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.172+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:07.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.172+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338074f40 con 0x7f8338069320 2021-03-09T10:30:07.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.172+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9184 cs=0 l=1 rev1=1 rx=0x7f8338042100 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:07.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.172+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8338069320 2021-03-09T10:30:07.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.172+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:30:07.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:07.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:30:07.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:30:07 smithi146 bash[13136]: debug 2021-03-09T10:30:07.172+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26633 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:30:12.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:30:12.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:12.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9184 cs=0 l=1 rev1=1 rx=0x7f8338042100 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:12.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9184 cs=0 l=1 rev1=1 rx=0x7f8338042100 tx=0x7f8344021000).stop 2021-03-09T10:30:12.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:12.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:12.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:12.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:12.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:12.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8324006200 2021-03-09T10:30:12.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:30:12.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ff60 con 0x7f8324008870 2021-03-09T10:30:12.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:12.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b930 con 0x7f8324005e00 2021-03-09T10:30:12.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:30:12.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:12.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:12.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:12.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:12.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:12.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:12.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380613d0 con 0x7f8324008870 2021-03-09T10:30:12.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fd0 secure :-1 s=READY pgs=9196 cs=0 l=1 rev1=1 rx=0x7f834403b5c0 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:12.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403b220 con 0x7f8324008870 2021-03-09T10:30:12.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8324008870 2021-03-09T10:30:12.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:12.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440173d0 con 0x7f8324008870 2021-03-09T10:30:12.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:30:12 smithi146 bash[13136]: debug 2021-03-09T10:30:12.176+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26659 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:30:17.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338009fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:30:17.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338009fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:17.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fd0 secure :-1 s=READY pgs=9196 cs=0 l=1 rev1=1 rx=0x7f834403b5c0 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:17.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fd0 secure :-1 s=READY pgs=9196 cs=0 l=1 rev1=1 rx=0x7f834403b5c0 tx=0x7f8344021000).stop 2021-03-09T10:30:17.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338009fd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:17.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009fd0 unknown :-1 s=CLOSED pgs=9196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:17.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:17.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.180+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:30:17.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.180+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:17.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.180+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:17.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.180+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:17.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012880 con 0x7f8324006200 2021-03-09T10:30:17.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c230 con 0x7f8338069320 2021-03-09T10:30:17.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bb80 con 0x7f8324005e00 2021-03-09T10:30:17.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.180+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:30:17.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.180+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:17.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.180+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:17.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.180+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:17.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.180+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:17.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.180+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:17.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.180+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:17.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.180+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014350 con 0x7f8338069320 2021-03-09T10:30:17.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.184+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9204 cs=0 l=1 rev1=1 rx=0x7f833c026520 tx=0x7f8344012fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:17.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa020 con 0x7f8338069320 2021-03-09T10:30:17.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338069320 2021-03-09T10:30:17.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:17.192+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338069320 2021-03-09T10:30:17.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:30:17 smithi146 bash[13136]: debug 2021-03-09T10:30:17.180+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26684 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:30:22.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:30:22.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:22.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9204 cs=0 l=1 rev1=1 rx=0x7f833c026520 tx=0x7f8344012fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:22.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9204 cs=0 l=1 rev1=1 rx=0x7f833c026520 tx=0x7f8344012fa0).stop 2021-03-09T10:30:22.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:22.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:22.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:22.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:22.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:22.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064390 con 0x7f8324006200 2021-03-09T10:30:22.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8324008870 2021-03-09T10:30:22.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ff60 con 0x7f8324005e00 2021-03-09T10:30:22.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:30:22.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:22.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:30:22.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:22.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:22.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:22.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:22.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:22.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:22.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338074f40 con 0x7f8324008870 2021-03-09T10:30:22.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.184+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 secure :-1 s=READY pgs=9210 cs=0 l=1 rev1=1 rx=0x7f834404c5a0 tx=0x7f8344021000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:22.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8324008870 2021-03-09T10:30:22.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8324008870 2021-03-09T10:30:22.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:22.192+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404e110 con 0x7f8324008870 2021-03-09T10:30:22.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:30:22 smithi146 bash[13136]: debug 2021-03-09T10:30:22.184+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26712 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:30:27.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338023140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:30:27.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338023140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:27.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 secure :-1 s=READY pgs=9210 cs=0 l=1 rev1=1 rx=0x7f834404c5a0 tx=0x7f8344021000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:27.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 secure :-1 s=READY pgs=9210 cs=0 l=1 rev1=1 rx=0x7f834404c5a0 tx=0x7f8344021000).stop 2021-03-09T10:30:27.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338023140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:27.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 unknown :-1 s=CLOSED pgs=9210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:27.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:27.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:27.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:27.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b930 con 0x7f8324006200 2021-03-09T10:30:27.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800bda0 con 0x7f8338069320 2021-03-09T10:30:27.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012880 con 0x7f8324005e00 2021-03-09T10:30:27.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:30:27.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:27.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:27.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:30:27.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:27.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:27.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:27.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:27.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:27.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380613d0 con 0x7f8338069320 2021-03-09T10:30:27.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.192+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9218 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:27.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.200+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334003550 con 0x7f8338069320 2021-03-09T10:30:27.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.200+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334060220 con 0x7f8338069320 2021-03-09T10:30:27.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:27.200+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d220 con 0x7f8338069320 2021-03-09T10:30:27.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:30:27 smithi146 bash[13136]: debug 2021-03-09T10:30:27.188+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26736 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:30:32.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:30:32.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:32.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9218 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:32.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9218 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334038000).stop 2021-03-09T10:30:32.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:32.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:32.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:32.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:32.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:32.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064950 con 0x7f8324006200 2021-03-09T10:30:32.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011b70 con 0x7f8324008870 2021-03-09T10:30:32.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014710 con 0x7f8324005e00 2021-03-09T10:30:32.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:30:32.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:32.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:32.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:30:32.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:32.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:32.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:32.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:32.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:32.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b8e0 con 0x7f8324008870 2021-03-09T10:30:32.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 secure :-1 s=READY pgs=9228 cs=0 l=1 rev1=1 rx=0x7f833802e2e0 tx=0x7f8344012fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:32.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8324008870 2021-03-09T10:30:32.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.196+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c026380 con 0x7f8324008870 2021-03-09T10:30:32.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:32.200+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f8324008870 2021-03-09T10:30:32.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:30:32 smithi146 bash[13136]: debug 2021-03-09T10:30:32.192+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26764 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:30:37.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338023140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:30:37.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338023140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:37.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 secure :-1 s=READY pgs=9228 cs=0 l=1 rev1=1 rx=0x7f833802e2e0 tx=0x7f8344012fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:37.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 secure :-1 s=READY pgs=9228 cs=0 l=1 rev1=1 rx=0x7f833802e2e0 tx=0x7f8344012fa0).stop 2021-03-09T10:30:37.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338023140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:37.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023140 unknown :-1 s=CLOSED pgs=9228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:37.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:37.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:37.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:30:37.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:37.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:30:37.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:37.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:37.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072f20 con 0x7f8324006200 2021-03-09T10:30:37.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8338069320 2021-03-09T10:30:37.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064390 con 0x7f8324005e00 2021-03-09T10:30:37.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:37.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:37.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:37.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:37.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:37.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380266a0 con 0x7f8338069320 2021-03-09T10:30:37.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.200+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9239 cs=0 l=1 rev1=1 rx=0x7f833c079870 tx=0x7f833c040dd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:37.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.204+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00c420 con 0x7f8338069320 2021-03-09T10:30:37.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.204+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c020ee0 con 0x7f8338069320 2021-03-09T10:30:37.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:37.208+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c022940 con 0x7f8338069320 2021-03-09T10:30:37.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:30:37 smithi146 bash[13136]: debug 2021-03-09T10:30:37.200+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26788 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:30:42.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:30:42.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:42.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9239 cs=0 l=1 rev1=1 rx=0x7f833c079870 tx=0x7f833c040dd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:42.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9239 cs=0 l=1 rev1=1 rx=0x7f833c079870 tx=0x7f833c040dd0).stop 2021-03-09T10:30:42.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:42.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:42.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:42.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:42.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:42.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:30:42.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:42.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:42.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ff60 con 0x7f8324006200 2021-03-09T10:30:42.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012880 con 0x7f8324008870 2021-03-09T10:30:42.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380070a0 con 0x7f8324005e00 2021-03-09T10:30:42.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:30:42.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:42.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:42.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:42.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:42.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:42.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801cc50 con 0x7f8324008870 2021-03-09T10:30:42.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.204+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f0c0 secure :-1 s=READY pgs=9245 cs=0 l=1 rev1=1 rx=0x7f833401ac50 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:42.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.208+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8324008870 2021-03-09T10:30:42.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.208+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401aa50 con 0x7f8324008870 2021-03-09T10:30:42.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:42.212+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340703c0 con 0x7f8324008870 2021-03-09T10:30:42.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:30:42 smithi146 bash[13136]: debug 2021-03-09T10:30:42.204+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26816 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:30:47.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:30:47.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:47.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f0c0 secure :-1 s=READY pgs=9245 cs=0 l=1 rev1=1 rx=0x7f833401ac50 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:47.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f0c0 secure :-1 s=READY pgs=9245 cs=0 l=1 rev1=1 rx=0x7f833401ac50 tx=0x7f8334038000).stop 2021-03-09T10:30:47.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801f0c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:47.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f0c0 unknown :-1 s=CLOSED pgs=9245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:47.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:47.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:47.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:47.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e760 con 0x7f8324006200 2021-03-09T10:30:47.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380164f0 con 0x7f8338069320 2021-03-09T10:30:47.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011b70 con 0x7f8324005e00 2021-03-09T10:30:47.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:30:47.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:47.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:30:47.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:47.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:47.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:47.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:47.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:47.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:47.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.208+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380283e0 con 0x7f8338069320 2021-03-09T10:30:47.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.212+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9256 cs=0 l=1 rev1=1 rx=0x7f834402c460 tx=0x7f834404ee10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:47.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.212+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8338069320 2021-03-09T10:30:47.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.212+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:30:47.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:47.216+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:30:47.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:30:47 smithi146 bash[13136]: debug 2021-03-09T10:30:47.212+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26840 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:30:52.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:30:52.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:52.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9256 cs=0 l=1 rev1=1 rx=0x7f834402c460 tx=0x7f834404ee10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:52.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9256 cs=0 l=1 rev1=1 rx=0x7f834402c460 tx=0x7f834404ee10).stop 2021-03-09T10:30:52.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:52.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:52.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:52.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:52.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:52.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:30:52.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014710 con 0x7f8324006200 2021-03-09T10:30:52.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:52.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022070 con 0x7f8324008870 2021-03-09T10:30:52.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ff60 con 0x7f8324005e00 2021-03-09T10:30:52.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:30:52.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:52.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:52.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:52.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:52.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:52.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:52.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014350 con 0x7f8324008870 2021-03-09T10:30:52.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9262 cs=0 l=1 rev1=1 rx=0x7f833c022210 tx=0x7f833c095000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:52.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c035370 con 0x7f8324008870 2021-03-09T10:30:52.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.216+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c062350 con 0x7f8324008870 2021-03-09T10:30:52.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:52.220+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0263f0 con 0x7f8324008870 2021-03-09T10:30:52.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:30:52 smithi146 bash[13136]: debug 2021-03-09T10:30:52.216+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26868 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:30:57.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:30:57.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:57.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9262 cs=0 l=1 rev1=1 rx=0x7f833c022210 tx=0x7f833c095000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:57.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9262 cs=0 l=1 rev1=1 rx=0x7f833c022210 tx=0x7f833c095000).stop 2021-03-09T10:30:57.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:57.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=CLOSED pgs=9262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:57.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:57.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:57.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:57.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b590 con 0x7f8324006200 2021-03-09T10:30:57.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8338069320 2021-03-09T10:30:57.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d5c0 con 0x7f8324005e00 2021-03-09T10:30:57.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:30:57.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:57.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:30:57.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:57.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:57.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:57.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:57.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:57.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:57.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b8e0 con 0x7f8338069320 2021-03-09T10:30:57.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9270 cs=0 l=1 rev1=1 rx=0x7f833c05e250 tx=0x7f833c095000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:57.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.219+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c062530 con 0x7f8338069320 2021-03-09T10:30:57.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c061300 con 0x7f8338069320 2021-03-09T10:30:57.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:30:57.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f8338069320 2021-03-09T10:30:57.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:30:57 smithi146 bash[13136]: debug 2021-03-09T10:30:57.220+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26892 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:31:02.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:31:02.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:02.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9270 cs=0 l=1 rev1=1 rx=0x7f833c05e250 tx=0x7f833c095000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:02.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9270 cs=0 l=1 rev1=1 rx=0x7f833c05e250 tx=0x7f833c095000).stop 2021-03-09T10:31:02.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:02.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:02.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:02.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:02.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:02.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802bb80 con 0x7f8324006200 2021-03-09T10:31:02.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380164f0 con 0x7f8324008870 2021-03-09T10:31:02.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011b70 con 0x7f8324005e00 2021-03-09T10:31:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:31:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:31:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:02.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.223+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:02.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.227+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:02.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.227+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:02.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.227+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.227+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.227+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380266a0 con 0x7f8324008870 2021-03-09T10:31:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.227+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9279 cs=0 l=1 rev1=1 rx=0x7f834405d960 tx=0x7f834404ee10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:02.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.227+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f8324008870 2021-03-09T10:31:02.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.227+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T10:31:02.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:02.231+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8324008870 2021-03-09T10:31:02.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:31:02 smithi146 bash[13136]: debug 2021-03-09T10:31:02.223+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26920 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:31:07.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.227+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:31:07.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.227+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:07.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.227+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9279 cs=0 l=1 rev1=1 rx=0x7f834405d960 tx=0x7f834404ee10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:07.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.227+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9279 cs=0 l=1 rev1=1 rx=0x7f834405d960 tx=0x7f834404ee10).stop 2021-03-09T10:31:07.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.227+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:07.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.227+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=CLOSED pgs=9279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:07.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.227+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:07.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.227+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:07.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.227+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:07.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.227+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012880 con 0x7f8324006200 2021-03-09T10:31:07.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.227+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ff60 con 0x7f8338069320 2021-03-09T10:31:07.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.227+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324005e00 2021-03-09T10:31:07.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.227+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:31:07.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.227+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:31:07.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.227+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:07.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.227+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:07.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.231+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:07.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.231+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:07.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.231+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:07.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.231+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:07.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.231+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:07.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.231+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380283e0 con 0x7f8338069320 2021-03-09T10:31:07.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.231+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9291 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:07.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.235+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404d380 con 0x7f8338069320 2021-03-09T10:31:07.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.235+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404d380 con 0x7f8338069320 2021-03-09T10:31:07.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:07.239+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334007860 con 0x7f8338069320 2021-03-09T10:31:07.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:31:07 smithi146 bash[13136]: debug 2021-03-09T10:31:07.231+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26944 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:31:12.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.231+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:31:12.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.231+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:12.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.231+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9291 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:12.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.231+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9291 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f8334038000).stop 2021-03-09T10:31:12.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.235+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:12.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.235+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:12.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.235+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:12.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.235+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:31:12.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.235+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:12.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.235+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:12.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.235+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:12.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.235+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064950 con 0x7f8324006200 2021-03-09T10:31:12.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.235+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380151c0 con 0x7f8324008870 2021-03-09T10:31:12.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.235+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064390 con 0x7f8324005e00 2021-03-09T10:31:12.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.235+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:31:12.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.235+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:12.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.235+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:12.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.239+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338005050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:12.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.239+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338005050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:12.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.239+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:12.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.239+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:12.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.239+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338074740 con 0x7f8324008870 2021-03-09T10:31:12.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.239+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9301 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c07ab80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:12.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.239+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c045540 con 0x7f8324008870 2021-03-09T10:31:12.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.243+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c045540 con 0x7f8324008870 2021-03-09T10:31:12.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:12.243+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054310 con 0x7f8324008870 2021-03-09T10:31:12.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:31:12 smithi146 bash[13136]: debug 2021-03-09T10:31:12.235+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26972 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:31:17.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:31:17.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:17.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9301 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c07ab80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:17.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9301 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c07ab80).stop 2021-03-09T10:31:17.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:17.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=CLOSED pgs=9301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:17.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:17.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:17.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:17.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8324006200 2021-03-09T10:31:17.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380164f0 con 0x7f8338069320 2021-03-09T10:31:17.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011b70 con 0x7f8324005e00 2021-03-09T10:31:17.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:31:17.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:17.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:31:17.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.239+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.243+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.243+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:17.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.243+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b8e0 con 0x7f8338069320 2021-03-09T10:31:17.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.243+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9305 cs=0 l=1 rev1=1 rx=0x7f833406d540 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:17.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.243+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340077b0 con 0x7f8338069320 2021-03-09T10:31:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.243+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334056380 con 0x7f8338069320 2021-03-09T10:31:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:17.247+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d220 con 0x7f8338069320 2021-03-09T10:31:17.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:31:17 smithi146 bash[13136]: debug 2021-03-09T10:31:17.239+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 26996 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:31:22.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:31:22.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:22.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9305 cs=0 l=1 rev1=1 rx=0x7f833406d540 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:22.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9305 cs=0 l=1 rev1=1 rx=0x7f833406d540 tx=0x7f8334038000).stop 2021-03-09T10:31:22.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:22.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:22.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:22.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:22.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:22.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ff60 con 0x7f8324006200 2021-03-09T10:31:22.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b1a0 con 0x7f8324008870 2021-03-09T10:31:22.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065d40 con 0x7f8324005e00 2021-03-09T10:31:22.246 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:31:22.246 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:22.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:31:22.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:22.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.243+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:22.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.247+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:22.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.247+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:22.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.247+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:22.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.247+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:22.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.247+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b510 con 0x7f8324008870 2021-03-09T10:31:22.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.247+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9316 cs=0 l=1 rev1=1 rx=0x7f8338064ed0 tx=0x7f834404ee10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:22.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.251+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f8324008870 2021-03-09T10:31:22.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.251+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T10:31:22.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:22.251+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8324008870 2021-03-09T10:31:22.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:31:22 smithi146 bash[13136]: debug 2021-03-09T10:31:22.243+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27024 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:31:27.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:31:27.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:27.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9316 cs=0 l=1 rev1=1 rx=0x7f8338064ed0 tx=0x7f834404ee10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:27.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9316 cs=0 l=1 rev1=1 rx=0x7f8338064ed0 tx=0x7f834404ee10).stop 2021-03-09T10:31:27.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:27.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=CLOSED pgs=9316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:27.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:27.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:27.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:27.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006200 2021-03-09T10:31:27.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d5c0 con 0x7f8338069320 2021-03-09T10:31:27.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064950 con 0x7f8324005e00 2021-03-09T10:31:27.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:31:27.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:31:27.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:27.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:27.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.247+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:27.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.251+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f8338004e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:27.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.251+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f8338004e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:27.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.251+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:27.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.251+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:27.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.251+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380266a0 con 0x7f8338069320 2021-03-09T10:31:27.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.251+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9322 cs=0 l=1 rev1=1 rx=0x7f83380245f0 tx=0x7f834404ee10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:27.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.251+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:31:27.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.251+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033080 con 0x7f8338069320 2021-03-09T10:31:27.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:27.255+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ba80 con 0x7f8338069320 2021-03-09T10:31:27.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:31:27 smithi146 bash[13136]: debug 2021-03-09T10:31:27.247+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27048 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:31:32.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.251+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:31:32.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.251+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:32.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.251+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9322 cs=0 l=1 rev1=1 rx=0x7f83380245f0 tx=0x7f834404ee10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:32.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.251+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9322 cs=0 l=1 rev1=1 rx=0x7f83380245f0 tx=0x7f834404ee10).stop 2021-03-09T10:31:32.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.251+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:32.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.251+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:32.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.251+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:32.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.251+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:32.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.251+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:32.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.251+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011b70 con 0x7f8324006200 2021-03-09T10:31:32.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.251+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ff60 con 0x7f8324008870 2021-03-09T10:31:32.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.251+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e860 con 0x7f8324005e00 2021-03-09T10:31:32.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.251+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801e0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:31:32.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.251+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801e0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:32.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.251+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:31:32.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.255+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:32.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.255+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:32.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.255+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801e0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:32.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.255+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801e0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:32.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.255+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:32.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.255+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:32.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.255+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380283e0 con 0x7f8324008870 2021-03-09T10:31:32.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.255+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801dbe0 secure :-1 s=READY pgs=9333 cs=0 l=1 rev1=1 rx=0x7f8338057a70 tx=0x7f834404ee10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:32.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.259+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440439c0 con 0x7f8324008870 2021-03-09T10:31:32.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.259+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T10:31:32.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:32.259+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440171c0 con 0x7f8324008870 2021-03-09T10:31:32.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:31:32 smithi146 bash[13136]: debug 2021-03-09T10:31:32.251+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27075 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:31:37.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.255+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:31:37.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.255+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:37.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.255+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801dbe0 secure :-1 s=READY pgs=9333 cs=0 l=1 rev1=1 rx=0x7f8338057a70 tx=0x7f834404ee10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:37.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.255+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801dbe0 secure :-1 s=READY pgs=9333 cs=0 l=1 rev1=1 rx=0x7f8338057a70 tx=0x7f834404ee10).stop 2021-03-09T10:31:37.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.255+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801dbe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:37.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.255+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801dbe0 unknown :-1 s=CLOSED pgs=9333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:37.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.255+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:37.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.255+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:37.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.255+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:37.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.255+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007390 con 0x7f8324006200 2021-03-09T10:31:37.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.255+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ff60 con 0x7f8338069320 2021-03-09T10:31:37.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.255+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801cd10 con 0x7f8324005e00 2021-03-09T10:31:37.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.259+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:31:37.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.259+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:37.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.259+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:37.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.259+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801e0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:31:37.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.259+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801e0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:37.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.259+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801e0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:37.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.259+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801e0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:37.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.259+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:37.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.259+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:37.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.259+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014350 con 0x7f8338069320 2021-03-09T10:31:37.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.259+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a2e0 secure :-1 s=READY pgs=9340 cs=0 l=1 rev1=1 rx=0x7f833402dcb0 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:37.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.263+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334000d30 con 0x7f8338069320 2021-03-09T10:31:37.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.263+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340336e0 con 0x7f8338069320 2021-03-09T10:31:37.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:37.267+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d220 con 0x7f8338069320 2021-03-09T10:31:37.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:31:37 smithi146 bash[13136]: debug 2021-03-09T10:31:37.259+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27100 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:31:42.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.259+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:31:42.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.259+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:42.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.259+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a2e0 secure :-1 s=READY pgs=9340 cs=0 l=1 rev1=1 rx=0x7f833402dcb0 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:42.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.259+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a2e0 secure :-1 s=READY pgs=9340 cs=0 l=1 rev1=1 rx=0x7f833402dcb0 tx=0x7f8334038000).stop 2021-03-09T10:31:42.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.259+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801a2e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:42.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.259+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a2e0 unknown :-1 s=CLOSED pgs=9340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:42.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.259+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:42.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.259+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:42.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.259+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:42.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.259+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d5c0 con 0x7f8324006200 2021-03-09T10:31:42.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.259+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064950 con 0x7f8324008870 2021-03-09T10:31:42.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.259+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011400 con 0x7f8324005e00 2021-03-09T10:31:42.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.259+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:31:42.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.259+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:42.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.263+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:31:42.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.263+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:42.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.263+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:42.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.263+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:42.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.263+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:42.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.263+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:42.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.263+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:42.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.263+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064d40 con 0x7f8324008870 2021-03-09T10:31:42.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.263+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017420 secure :-1 s=READY pgs=9348 cs=0 l=1 rev1=1 rx=0x7f833806bee0 tx=0x7f834404ee10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:42.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.267+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400fef0 con 0x7f8324008870 2021-03-09T10:31:42.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.267+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400fef0 con 0x7f8324008870 2021-03-09T10:31:42.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:42.271+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8324008870 2021-03-09T10:31:42.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:31:42 smithi146 bash[13136]: debug 2021-03-09T10:31:42.263+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27126 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:31:47.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338017420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:31:47.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338017420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:47.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017420 secure :-1 s=READY pgs=9348 cs=0 l=1 rev1=1 rx=0x7f833806bee0 tx=0x7f834404ee10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:47.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017420 secure :-1 s=READY pgs=9348 cs=0 l=1 rev1=1 rx=0x7f833806bee0 tx=0x7f834404ee10).stop 2021-03-09T10:31:47.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338017420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:47.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017420 unknown :-1 s=CLOSED pgs=9348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:47.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:47.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:47.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:47.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8324006200 2021-03-09T10:31:47.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007390 con 0x7f8338069320 2021-03-09T10:31:47.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380166f0 con 0x7f8324005e00 2021-03-09T10:31:47.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:31:47.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:47.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:31:47.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:47.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:47.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:47.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:47.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:47.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:47.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.266+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b8e0 con 0x7f8338069320 2021-03-09T10:31:47.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.270+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9357 cs=0 l=1 rev1=1 rx=0x7f8334009bd0 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:47.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.270+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8338069320 2021-03-09T10:31:47.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.270+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000d10 con 0x7f8338069320 2021-03-09T10:31:47.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:47.274+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334033870 con 0x7f8338069320 2021-03-09T10:31:47.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:31:47 smithi146 bash[13136]: debug 2021-03-09T10:31:47.267+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27151 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:31:52.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:31:52.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:52.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9357 cs=0 l=1 rev1=1 rx=0x7f8334009bd0 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:52.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9357 cs=0 l=1 rev1=1 rx=0x7f8334009bd0 tx=0x7f8334038000).stop 2021-03-09T10:31:52.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:52.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:52.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:52.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:52.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:52.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:31:52.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d5c0 con 0x7f8324006200 2021-03-09T10:31:52.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:52.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e860 con 0x7f8324008870 2021-03-09T10:31:52.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c360 con 0x7f8324005e00 2021-03-09T10:31:52.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:52.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:31:52.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:52.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:52.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:52.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:52.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:52.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800d420 con 0x7f8324008870 2021-03-09T10:31:52.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.270+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017420 secure :-1 s=READY pgs=9366 cs=0 l=1 rev1=1 rx=0x7f833c0364e0 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:52.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.274+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0629a0 con 0x7f8324008870 2021-03-09T10:31:52.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.274+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f8324008870 2021-03-09T10:31:52.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:52.278+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8324008870 2021-03-09T10:31:52.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:31:52 smithi146 bash[13136]: debug 2021-03-09T10:31:52.271+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27177 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:31:57.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338017420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:31:57.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338017420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:57.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017420 secure :-1 s=READY pgs=9366 cs=0 l=1 rev1=1 rx=0x7f833c0364e0 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:57.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017420 secure :-1 s=READY pgs=9366 cs=0 l=1 rev1=1 rx=0x7f833c0364e0 tx=0x7f833c09a310).stop 2021-03-09T10:31:57.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338017420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:57.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017420 unknown :-1 s=CLOSED pgs=9366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:57.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:57.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380181c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:57.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380181c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:57.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:57.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380747c0 con 0x7f8324006200 2021-03-09T10:31:57.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007390 con 0x7f8338069320 2021-03-09T10:31:57.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c230 con 0x7f8324005e00 2021-03-09T10:31:57.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:31:57.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:31:57.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:57.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.274+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:57.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.278+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:57.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.278+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:57.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.278+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:57.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.278+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:57.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.278+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065400 con 0x7f8338069320 2021-03-09T10:31:57.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.278+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380181c0 secure :-1 s=READY pgs=9375 cs=0 l=1 rev1=1 rx=0x7f8344042be0 tx=0x7f834404ee10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:57.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.278+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f8338069320 2021-03-09T10:31:57.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.282+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:31:57.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:31:57.282+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:31:57.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:31:57 smithi146 bash[13136]: debug 2021-03-09T10:31:57.279+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27203 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:32:02.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.278+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380181c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:32:02.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.278+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380181c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:02.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.278+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380181c0 secure :-1 s=READY pgs=9375 cs=0 l=1 rev1=1 rx=0x7f8344042be0 tx=0x7f834404ee10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:02.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.278+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380181c0 secure :-1 s=READY pgs=9375 cs=0 l=1 rev1=1 rx=0x7f8344042be0 tx=0x7f834404ee10).stop 2021-03-09T10:32:02.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.278+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f83380181c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:02.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.278+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f83380181c0 unknown :-1 s=CLOSED pgs=9375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:02.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.278+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:02.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.278+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:02.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.278+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:02.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.278+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8324006200 2021-03-09T10:32:02.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.278+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007390 con 0x7f8324008870 2021-03-09T10:32:02.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.278+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a420 con 0x7f8324005e00 2021-03-09T10:32:02.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.278+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:32:02.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.278+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:02.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.278+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:32:02.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.278+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:02.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.282+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:02.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.282+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:02.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.282+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:02.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.282+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:02.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.282+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:02.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.282+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062000 con 0x7f8324008870 2021-03-09T10:32:02.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.282+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801abf0 secure :-1 s=READY pgs=9383 cs=0 l=1 rev1=1 rx=0x7f833c0616a0 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:02.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.286+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c061940 con 0x7f8324008870 2021-03-09T10:32:02.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.286+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c061710 con 0x7f8324008870 2021-03-09T10:32:02.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:02.290+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c061bd0 con 0x7f8324008870 2021-03-09T10:32:02.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:32:02 smithi146 bash[13136]: debug 2021-03-09T10:32:02.283+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27229 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:32:07.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.282+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:32:07.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.282+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:07.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.282+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801abf0 secure :-1 s=READY pgs=9383 cs=0 l=1 rev1=1 rx=0x7f833c0616a0 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:07.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.282+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801abf0 secure :-1 s=READY pgs=9383 cs=0 l=1 rev1=1 rx=0x7f833c0616a0 tx=0x7f833c09a310).stop 2021-03-09T10:32:07.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.282+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801abf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:07.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.282+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801abf0 unknown :-1 s=CLOSED pgs=9383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:07.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.282+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:07.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.282+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:07.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.282+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:07.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.282+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c230 con 0x7f8324006200 2021-03-09T10:32:07.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.282+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f8338069320 2021-03-09T10:32:07.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.282+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380747c0 con 0x7f8324005e00 2021-03-09T10:32:07.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.286+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:32:07.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.286+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:07.289 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.286+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:07.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.286+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:32:07.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.286+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:07.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.286+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:07.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.286+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:07.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.286+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:07.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.286+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:07.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.286+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338008960 con 0x7f8338069320 2021-03-09T10:32:07.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.286+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9392 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:07.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.290+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8338069320 2021-03-09T10:32:07.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.290+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404e220 con 0x7f8338069320 2021-03-09T10:32:07.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:07.294+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334056990 con 0x7f8338069320 2021-03-09T10:32:07.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:32:07 smithi146 bash[13136]: debug 2021-03-09T10:32:07.287+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27255 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:32:12.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:32:12.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:12.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9392 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:12.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9392 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334038000).stop 2021-03-09T10:32:12.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:12.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:12.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:12.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:12.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018890 con 0x7f8324006200 2021-03-09T10:32:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380747c0 con 0x7f8324008870 2021-03-09T10:32:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380730a0 con 0x7f8324005e00 2021-03-09T10:32:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:32:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:32:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:12.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:12.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.290+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b8e0 con 0x7f8324008870 2021-03-09T10:32:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.294+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801abf0 secure :-1 s=READY pgs=9400 cs=0 l=1 rev1=1 rx=0x7f833400cfe0 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.298+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401cec0 con 0x7f8324008870 2021-03-09T10:32:12.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.298+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403a830 con 0x7f8324008870 2021-03-09T10:32:12.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:12.302+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403aac0 con 0x7f8324008870 2021-03-09T10:32:12.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:32:12 smithi146 bash[13136]: debug 2021-03-09T10:32:12.290+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27277 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:32:17.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:32:17.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:17.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801abf0 secure :-1 s=READY pgs=9400 cs=0 l=1 rev1=1 rx=0x7f833400cfe0 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:17.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801abf0 secure :-1 s=READY pgs=9400 cs=0 l=1 rev1=1 rx=0x7f833400cfe0 tx=0x7f8334038000).stop 2021-03-09T10:32:17.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801abf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:17.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801abf0 unknown :-1 s=CLOSED pgs=9400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:17.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:17.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:17.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:17.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a420 con 0x7f8324006200 2021-03-09T10:32:17.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c230 con 0x7f8338069320 2021-03-09T10:32:17.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dde0 con 0x7f8324005e00 2021-03-09T10:32:17.298 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:32:17.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:17.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:32:17.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:17.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.294+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:17.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.298+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:17.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.298+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:17.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.298+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:17.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.298+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:17.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.298+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024560 con 0x7f8338069320 2021-03-09T10:32:17.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.298+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9413 cs=0 l=1 rev1=1 rx=0x7f83380107e0 tx=0x7f834404ee10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:17.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.302+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:32:17.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.306+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338069320 2021-03-09T10:32:17.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:17.306+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:32:17.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:32:17 smithi146 bash[13136]: debug 2021-03-09T10:32:17.298+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27307 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:32:22.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.298+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:32:22.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.298+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:22.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.298+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9413 cs=0 l=1 rev1=1 rx=0x7f83380107e0 tx=0x7f834404ee10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:22.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.298+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9413 cs=0 l=1 rev1=1 rx=0x7f83380107e0 tx=0x7f834404ee10).stop 2021-03-09T10:32:22.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.298+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:22.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.298+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:22.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.298+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:22.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.298+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:22.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.298+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:22.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.298+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8324006200 2021-03-09T10:32:22.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.298+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d7c0 con 0x7f8324008870 2021-03-09T10:32:22.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.298+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014710 con 0x7f8324005e00 2021-03-09T10:32:22.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.298+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:32:22.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.298+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:22.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.302+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:32:22.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.302+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:22.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.302+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:22.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.302+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801dfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:22.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.302+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:22.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.302+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:22.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.302+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:22.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.302+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380667b0 con 0x7f8324008870 2021-03-09T10:32:22.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.302+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 secure :-1 s=READY pgs=9420 cs=0 l=1 rev1=1 rx=0x7f834402c9c0 tx=0x7f834404ee10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:22.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.306+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344079bb0 con 0x7f8324008870 2021-03-09T10:32:22.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.306+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T10:32:22.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:22.310+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402fba0 con 0x7f8324008870 2021-03-09T10:32:22.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:32:22 smithi146 bash[13136]: debug 2021-03-09T10:32:22.302+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27332 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:32:27.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.302+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:32:27.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.302+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:27.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.302+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 secure :-1 s=READY pgs=9420 cs=0 l=1 rev1=1 rx=0x7f834402c9c0 tx=0x7f834404ee10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:27.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 secure :-1 s=READY pgs=9420 cs=0 l=1 rev1=1 rx=0x7f834402c9c0 tx=0x7f834404ee10).stop 2021-03-09T10:32:27.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:27.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 unknown :-1 s=CLOSED pgs=9420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:27.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:27.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:27.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:27.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d7c0 con 0x7f8324006200 2021-03-09T10:32:27.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c230 con 0x7f8338069320 2021-03-09T10:32:27.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024070 con 0x7f8324005e00 2021-03-09T10:32:27.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:32:27.309 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:27.309 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:27.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:32:27.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:27.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:27.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:27.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:27.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:27.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017b00 con 0x7f8338069320 2021-03-09T10:32:27.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.306+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9426 cs=0 l=1 rev1=1 rx=0x7f833800d700 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:27.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.310+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338069320 2021-03-09T10:32:27.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.310+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338069320 2021-03-09T10:32:27.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:27.314+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338069320 2021-03-09T10:32:27.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:32:27 smithi146 bash[13136]: debug 2021-03-09T10:32:27.306+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27359 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:32:32.310 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:32:32.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:32.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9426 cs=0 l=1 rev1=1 rx=0x7f833800d700 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:32.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9426 cs=0 l=1 rev1=1 rx=0x7f833800d700 tx=0x7f833c09a310).stop 2021-03-09T10:32:32.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:32.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:32.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:32.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:32.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:32.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014710 con 0x7f8324006200 2021-03-09T10:32:32.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8324008870 2021-03-09T10:32:32.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801c230 con 0x7f8324005e00 2021-03-09T10:32:32.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:32:32.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:32.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:32:32.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:32.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.310+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:32.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.314+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:32.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.314+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:32.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.314+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:32.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.314+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:32.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.314+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062000 con 0x7f8324008870 2021-03-09T10:32:32.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.314+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 secure :-1 s=READY pgs=9435 cs=0 l=1 rev1=1 rx=0x7f8338054950 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:32.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.314+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa040 con 0x7f8324008870 2021-03-09T10:32:32.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.314+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00c430 con 0x7f8324008870 2021-03-09T10:32:32.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:32.318+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f8324008870 2021-03-09T10:32:32.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:32:32 smithi146 bash[13136]: debug 2021-03-09T10:32:32.310+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27385 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:32:37.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.313+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:32:37.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.313+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:37.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.313+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 secure :-1 s=READY pgs=9435 cs=0 l=1 rev1=1 rx=0x7f8338054950 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:37.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.313+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 secure :-1 s=READY pgs=9435 cs=0 l=1 rev1=1 rx=0x7f8338054950 tx=0x7f833c09a310).stop 2021-03-09T10:32:37.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.313+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:37.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.313+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 unknown :-1 s=CLOSED pgs=9435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:37.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.313+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:37.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.313+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:32:37.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.313+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:37.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.313+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:37.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.313+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:32:37.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.313+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:37.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.313+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:37.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.313+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dde0 con 0x7f8324006200 2021-03-09T10:32:37.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.313+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8338069320 2021-03-09T10:32:37.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.313+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380743e0 con 0x7f8324005e00 2021-03-09T10:32:37.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.317+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:37.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.317+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:37.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.317+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:37.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.317+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:37.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.317+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:37.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.317+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801d5c0 con 0x7f8338069320 2021-03-09T10:32:37.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.317+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9444 cs=0 l=1 rev1=1 rx=0x7f833c01cd40 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:37.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.321+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338069320 2021-03-09T10:32:37.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.321+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c020720 con 0x7f8338069320 2021-03-09T10:32:37.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:37.321+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c026d40 con 0x7f8338069320 2021-03-09T10:32:37.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:32:37 smithi146 bash[13136]: debug 2021-03-09T10:32:37.318+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27411 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:32:42.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:32:42.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:42.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9444 cs=0 l=1 rev1=1 rx=0x7f833c01cd40 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:42.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9444 cs=0 l=1 rev1=1 rx=0x7f833c01cd40 tx=0x7f833c09a310).stop 2021-03-09T10:32:42.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:42.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:42.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:42.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:42.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:42.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e860 con 0x7f8324006200 2021-03-09T10:32:42.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380743e0 con 0x7f8324008870 2021-03-09T10:32:42.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380736b0 con 0x7f8324005e00 2021-03-09T10:32:42.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:32:42.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:42.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:32:42.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:42.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:42.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:42.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:42.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:42.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:42.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8324008870 2021-03-09T10:32:42.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.321+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802e040 secure :-1 s=READY pgs=9452 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:42.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.325+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334027d70 con 0x7f8324008870 2021-03-09T10:32:42.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.325+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334027d70 con 0x7f8324008870 2021-03-09T10:32:42.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:42.329+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401c2a0 con 0x7f8324008870 2021-03-09T10:32:42.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:32:42 smithi146 bash[13136]: debug 2021-03-09T10:32:42.322+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27437 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:32:47.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833802e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:32:47.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833802e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:47.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802e040 secure :-1 s=READY pgs=9452 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:47.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802e040 secure :-1 s=READY pgs=9452 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f8334038000).stop 2021-03-09T10:32:47.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833802e040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:47.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802e040 unknown :-1 s=CLOSED pgs=9452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:47.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:47.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:47.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:47.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380736b0 con 0x7f8324006200 2021-03-09T10:32:47.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e0a0 con 0x7f8338069320 2021-03-09T10:32:47.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cf70 con 0x7f8324005e00 2021-03-09T10:32:47.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:32:47.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:32:47.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:47.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:47.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:47.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:47.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:47.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:47.333 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:47.333 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.325+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b8e0 con 0x7f8338069320 2021-03-09T10:32:47.333 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.329+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9461 cs=0 l=1 rev1=1 rx=0x7f8344054ec0 tx=0x7f834404ee10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:47.333 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.329+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f8338069320 2021-03-09T10:32:47.334 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.329+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033080 con 0x7f8338069320 2021-03-09T10:32:47.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:47.333+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:32:47.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:32:47 smithi146 bash[13136]: debug 2021-03-09T10:32:47.326+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27463 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:32:52.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:32:52.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:52.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9461 cs=0 l=1 rev1=1 rx=0x7f8344054ec0 tx=0x7f834404ee10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:52.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9461 cs=0 l=1 rev1=1 rx=0x7f8344054ec0 tx=0x7f834404ee10).stop 2021-03-09T10:32:52.333 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:52.333 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:52.333 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:52.334 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:52.334 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:32:52.334 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:52.334 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:52.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e860 con 0x7f8324006200 2021-03-09T10:32:52.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380743e0 con 0x7f8324008870 2021-03-09T10:32:52.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380736b0 con 0x7f8324005e00 2021-03-09T10:32:52.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:52.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:32:52.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.329+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:52.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.333+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:52.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.333+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:52.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.333+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:52.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.333+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:52.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.333+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338069170 con 0x7f8324008870 2021-03-09T10:32:52.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.333+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 secure :-1 s=READY pgs=9470 cs=0 l=1 rev1=1 rx=0x7f8334034050 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:52.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.337+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340261b0 con 0x7f8324008870 2021-03-09T10:32:52.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.337+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334031130 con 0x7f8324008870 2021-03-09T10:32:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:52.341+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340077b0 con 0x7f8324008870 2021-03-09T10:32:52.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:32:52 smithi146 bash[13136]: debug 2021-03-09T10:32:52.334+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27489 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:32:57.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:32:57.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:57.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 secure :-1 s=READY pgs=9470 cs=0 l=1 rev1=1 rx=0x7f8334034050 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:57.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 secure :-1 s=READY pgs=9470 cs=0 l=1 rev1=1 rx=0x7f8334034050 tx=0x7f8334038000).stop 2021-03-09T10:32:57.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380720f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:57.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 unknown :-1 s=CLOSED pgs=9470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:57.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:57.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:57.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:57.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cf70 con 0x7f8324006200 2021-03-09T10:32:57.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dde0 con 0x7f8338069320 2021-03-09T10:32:57.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8324005e00 2021-03-09T10:32:57.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:32:57.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:57.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:32:57.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:57.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:57.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:57.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:57.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:57.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:57.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801fc00 con 0x7f8338069320 2021-03-09T10:32:57.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.337+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9480 cs=0 l=1 rev1=1 rx=0x7f833401c870 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:57.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.341+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334047e50 con 0x7f8338069320 2021-03-09T10:32:57.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.341+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334015840 con 0x7f8338069320 2021-03-09T10:32:57.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:32:57.345+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334047ba0 con 0x7f8338069320 2021-03-09T10:32:57.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:32:57 smithi146 bash[13136]: debug 2021-03-09T10:32:57.338+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27515 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:33:02.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.337+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:33:02.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.337+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:02.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.337+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9480 cs=0 l=1 rev1=1 rx=0x7f833401c870 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:02.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.337+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9480 cs=0 l=1 rev1=1 rx=0x7f833401c870 tx=0x7f8334038000).stop 2021-03-09T10:33:02.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.337+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:02.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.337+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:02.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:02.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:02.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:02.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064020 con 0x7f8324006200 2021-03-09T10:33:02.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e0a0 con 0x7f8324008870 2021-03-09T10:33:02.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380736b0 con 0x7f8324005e00 2021-03-09T10:33:02.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:33:02.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:02.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:02.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:33:02.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:02.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:02.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:02.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:02.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:02.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8324008870 2021-03-09T10:33:02.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.341+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074d80 secure :-1 s=READY pgs=9487 cs=0 l=1 rev1=1 rx=0x7f833800c110 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:02.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.345+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8324008870 2021-03-09T10:33:02.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.345+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0a1dd0 con 0x7f8324008870 2021-03-09T10:33:02.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:02.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0a1970 con 0x7f8324008870 2021-03-09T10:33:02.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:33:02 smithi146 bash[13136]: debug 2021-03-09T10:33:02.342+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27541 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:33:07.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338074d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:33:07.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338074d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:07.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074d80 secure :-1 s=READY pgs=9487 cs=0 l=1 rev1=1 rx=0x7f833800c110 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:07.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074d80 secure :-1 s=READY pgs=9487 cs=0 l=1 rev1=1 rx=0x7f833800c110 tx=0x7f833c09a310).stop 2021-03-09T10:33:07.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338074d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:07.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074d80 unknown :-1 s=CLOSED pgs=9487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:07.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:07.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:07.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:07.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380732e0 con 0x7f8324006200 2021-03-09T10:33:07.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a90 con 0x7f8338069320 2021-03-09T10:33:07.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324005e00 2021-03-09T10:33:07.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:33:07.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:07.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:33:07.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:07.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:07.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:07.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:07.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:07.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:07.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b8e0 con 0x7f8338069320 2021-03-09T10:33:07.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.345+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9496 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f834404ee10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:07.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8338069320 2021-03-09T10:33:07.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f8338069320 2021-03-09T10:33:07.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:07.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338069320 2021-03-09T10:33:07.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:33:07 smithi146 bash[13136]: debug 2021-03-09T10:33:07.346+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27567 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:33:12.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:33:12.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:12.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9496 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f834404ee10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:12.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9496 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f834404ee10).stop 2021-03-09T10:33:12.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:12.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:12.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:12.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:12.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:12.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015bf0 con 0x7f8324006200 2021-03-09T10:33:12.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324008870 2021-03-09T10:33:12.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a0e0 con 0x7f8324005e00 2021-03-09T10:33:12.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:33:12.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:33:12.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:12.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:12.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.349+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:12.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.353+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:12.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.353+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:12.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.353+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:12.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.353+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:12.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.353+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8324008870 2021-03-09T10:33:12.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.353+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 secure :-1 s=READY pgs=9504 cs=0 l=1 rev1=1 rx=0x7f833401ae30 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:12.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334015840 con 0x7f8324008870 2021-03-09T10:33:12.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833406d180 con 0x7f8324008870 2021-03-09T10:33:12.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:12.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340151c0 con 0x7f8324008870 2021-03-09T10:33:12.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:33:12 smithi146 bash[13136]: debug 2021-03-09T10:33:12.354+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27593 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:33:17.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:33:17.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:17.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 secure :-1 s=READY pgs=9504 cs=0 l=1 rev1=1 rx=0x7f833401ae30 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:17.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 secure :-1 s=READY pgs=9504 cs=0 l=1 rev1=1 rx=0x7f833401ae30 tx=0x7f8334038000).stop 2021-03-09T10:33:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380720f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 unknown :-1 s=CLOSED pgs=9504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:17.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380736b0 con 0x7f8324006200 2021-03-09T10:33:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007730 con 0x7f8338069320 2021-03-09T10:33:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324005e00 2021-03-09T10:33:17.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:33:17.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:17.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:33:17.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:17.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:17.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:17.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338025180 con 0x7f8338069320 2021-03-09T10:33:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.357+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9513 cs=0 l=1 rev1=1 rx=0x7f833403c050 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:17.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.361+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334015480 con 0x7f8338069320 2021-03-09T10:33:17.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.361+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402d6a0 con 0x7f8338069320 2021-03-09T10:33:17.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:17.361+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338069320 2021-03-09T10:33:17.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:33:17 smithi146 bash[13136]: debug 2021-03-09T10:33:17.358+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27619 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:33:22.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:33:22.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:22.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9513 cs=0 l=1 rev1=1 rx=0x7f833403c050 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:22.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9513 cs=0 l=1 rev1=1 rx=0x7f833403c050 tx=0x7f8334038000).stop 2021-03-09T10:33:22.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:22.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:22.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:22.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:22.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338064b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:22.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009920 con 0x7f8324006200 2021-03-09T10:33:22.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e860 con 0x7f8324008870 2021-03-09T10:33:22.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024070 con 0x7f8324005e00 2021-03-09T10:33:22.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:33:22.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:22.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:33:22.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:22.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338064b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:22.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:22.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:22.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:22.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:22.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010650 con 0x7f8324008870 2021-03-09T10:33:22.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.361+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338064b20 secure :-1 s=READY pgs=9522 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:22.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa040 con 0x7f8324008870 2021-03-09T10:33:22.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c035370 con 0x7f8324008870 2021-03-09T10:33:22.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:22.369+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8324008870 2021-03-09T10:33:22.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:33:22 smithi146 bash[13136]: debug 2021-03-09T10:33:22.362+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27645 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:33:27.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338064b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:33:27.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338064b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:27.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338064b20 secure :-1 s=READY pgs=9522 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:27.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338064b20 secure :-1 s=READY pgs=9522 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c09a310).stop 2021-03-09T10:33:27.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338064b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:27.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338064b20 unknown :-1 s=CLOSED pgs=9522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:27.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:27.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:27.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:27.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a0e0 con 0x7f8324006200 2021-03-09T10:33:27.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a90 con 0x7f8338069320 2021-03-09T10:33:27.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380736b0 con 0x7f8324005e00 2021-03-09T10:33:27.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:33:27.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:27.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:27.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:33:27.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:27.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:27.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:27.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:27.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:27.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.364+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b8e0 con 0x7f8338069320 2021-03-09T10:33:27.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.368+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9530 cs=0 l=1 rev1=1 rx=0x7f8334015fa0 tx=0x7f8334038000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:27.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.368+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334058e50 con 0x7f8338069320 2021-03-09T10:33:27.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.368+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404eec0 con 0x7f8338069320 2021-03-09T10:33:27.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:27.372+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833406d3d0 con 0x7f8338069320 2021-03-09T10:33:27.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:33:27 smithi146 bash[13136]: debug 2021-03-09T10:33:27.365+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27671 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:33:32.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:33:32.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:32.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9530 cs=0 l=1 rev1=1 rx=0x7f8334015fa0 tx=0x7f8334038000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:32.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9530 cs=0 l=1 rev1=1 rx=0x7f8334015fa0 tx=0x7f8334038000).stop 2021-03-09T10:33:32.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:32.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:32.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:32.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:32.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:32.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:33:32.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324006200 2021-03-09T10:33:32.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e860 con 0x7f8324008870 2021-03-09T10:33:32.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b860 con 0x7f8324005e00 2021-03-09T10:33:32.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:32.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:33:32.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:32.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:32.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:32.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:32.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:32.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:32.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.368+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e840 con 0x7f8324008870 2021-03-09T10:33:32.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.372+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801c430 secure :-1 s=READY pgs=9539 cs=0 l=1 rev1=1 rx=0x7f833c042950 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:32.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.372+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa040 con 0x7f8324008870 2021-03-09T10:33:32.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.376+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c061300 con 0x7f8324008870 2021-03-09T10:33:32.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:32.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c042380 con 0x7f8324008870 2021-03-09T10:33:32.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:33:32 smithi146 bash[13136]: debug 2021-03-09T10:33:32.373+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27697 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:33:37.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:33:37.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:37.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801c430 secure :-1 s=READY pgs=9539 cs=0 l=1 rev1=1 rx=0x7f833c042950 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:37.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801c430 secure :-1 s=READY pgs=9539 cs=0 l=1 rev1=1 rx=0x7f833c042950 tx=0x7f833c09a310).stop 2021-03-09T10:33:37.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:37.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801c430 unknown :-1 s=CLOSED pgs=9539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:37.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:37.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:37.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:37.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380736b0 con 0x7f8324006200 2021-03-09T10:33:37.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009fb0 con 0x7f8338069320 2021-03-09T10:33:37.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069fe0 con 0x7f8324005e00 2021-03-09T10:33:37.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:33:37.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:33:37.380 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:37.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:37.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:37.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:37.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:37.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:37.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:37.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.376+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010650 con 0x7f8338069320 2021-03-09T10:33:37.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9552 cs=0 l=1 rev1=1 rx=0x7f833c040c60 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:37.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338069320 2021-03-09T10:33:37.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c061510 con 0x7f8338069320 2021-03-09T10:33:37.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:37.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c042380 con 0x7f8338069320 2021-03-09T10:33:37.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:33:37 smithi146 bash[13136]: debug 2021-03-09T10:33:37.377+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27723 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:33:42.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:33:42.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:42.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9552 cs=0 l=1 rev1=1 rx=0x7f833c040c60 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:42.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9552 cs=0 l=1 rev1=1 rx=0x7f833c040c60 tx=0x7f833c09a310).stop 2021-03-09T10:33:42.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:42.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:42.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:42.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:42.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:42.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a90 con 0x7f8324006200 2021-03-09T10:33:42.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380736b0 con 0x7f8324008870 2021-03-09T10:33:42.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015bf0 con 0x7f8324005e00 2021-03-09T10:33:42.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:33:42.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:42.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:33:42.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:42.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:42.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.388+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:42.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:42.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.388+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:42.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:42.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.388+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8324008870 2021-03-09T10:33:42.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9559 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:42.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8324008870 2021-03-09T10:33:42.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0629a0 con 0x7f8324008870 2021-03-09T10:33:42.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:42.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f8324008870 2021-03-09T10:33:42.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:33:42 smithi146 bash[13136]: debug 2021-03-09T10:33:42.381+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27749 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:33:47.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:33:47.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:47.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9559 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:47.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9559 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c09a310).stop 2021-03-09T10:33:47.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:47.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=CLOSED pgs=9559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:47.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:47.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:47.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:47.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009fb0 con 0x7f8324006200 2021-03-09T10:33:47.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015bf0 con 0x7f8338069320 2021-03-09T10:33:47.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380124c0 con 0x7f8324005e00 2021-03-09T10:33:47.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:33:47.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:47.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:33:47.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:47.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.384+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:47.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.388+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:47.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:47.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.388+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:47.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:47.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.388+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801b8e0 con 0x7f8338069320 2021-03-09T10:33:47.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9565 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:47.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0629a0 con 0x7f8338069320 2021-03-09T10:33:47.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0629a0 con 0x7f8338069320 2021-03-09T10:33:47.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:47.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0792c0 con 0x7f8338069320 2021-03-09T10:33:47.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:33:47 smithi146 bash[13136]: debug 2021-03-09T10:33:47.389+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27774 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:33:52.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.388+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:33:52.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.388+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:52.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9565 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:52.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9565 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c09a310).stop 2021-03-09T10:33:52.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:52.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.388+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:52.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.388+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:52.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.388+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:52.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.388+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338061aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:52.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069fe0 con 0x7f8324006200 2021-03-09T10:33:52.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380124c0 con 0x7f8324008870 2021-03-09T10:33:52.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d130 con 0x7f8324005e00 2021-03-09T10:33:52.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.388+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:33:52.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.388+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:52.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.392+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:33:52.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.392+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:52.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338061aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:52.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:52.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:52.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:52.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:52.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806cef0 con 0x7f8324008870 2021-03-09T10:33:52.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338061aa0 secure :-1 s=READY pgs=9574 cs=0 l=1 rev1=1 rx=0x7f8338028e30 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:52.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c093a90 con 0x7f8324008870 2021-03-09T10:33:52.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01b7d0 con 0x7f8324008870 2021-03-09T10:33:52.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:52.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c093880 con 0x7f8324008870 2021-03-09T10:33:52.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:33:52 smithi146 bash[13136]: debug 2021-03-09T10:33:52.393+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27801 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:33:57.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338061aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:33:57.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338061aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:57.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338061aa0 secure :-1 s=READY pgs=9574 cs=0 l=1 rev1=1 rx=0x7f8338028e30 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:57.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338061aa0 secure :-1 s=READY pgs=9574 cs=0 l=1 rev1=1 rx=0x7f8338028e30 tx=0x7f833c09a310).stop 2021-03-09T10:33:57.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338061aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:57.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338061aa0 unknown :-1 s=CLOSED pgs=9574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:57.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:57.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:57.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:33:57.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:57.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:33:57.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:57.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:57.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d130 con 0x7f8324006200 2021-03-09T10:33:57.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8338069320 2021-03-09T10:33:57.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069fe0 con 0x7f8324005e00 2021-03-09T10:33:57.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:57.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:57.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:57.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:57.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:57.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.396+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8338069320 2021-03-09T10:33:57.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9584 cs=0 l=1 rev1=1 rx=0x7f833c0641a0 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:57.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0367b0 con 0x7f8338069320 2021-03-09T10:33:57.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0367b0 con 0x7f8338069320 2021-03-09T10:33:57.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:33:57.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c093960 con 0x7f8338069320 2021-03-09T10:33:57.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:33:57 smithi146 bash[13136]: debug 2021-03-09T10:33:57.397+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27826 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:34:02.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:34:02.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:02.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9584 cs=0 l=1 rev1=1 rx=0x7f833c0641a0 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:02.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9584 cs=0 l=1 rev1=1 rx=0x7f833c0641a0 tx=0x7f833c09a310).stop 2021-03-09T10:34:02.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:02.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:02.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:02.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:02.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.400+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:02.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324006200 2021-03-09T10:34:02.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f8324008870 2021-03-09T10:34:02.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380124c0 con 0x7f8324005e00 2021-03-09T10:34:02.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.404+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:34:02.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.404+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:02.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.404+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:34:02.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.404+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:02.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:02.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:02.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:02.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:02.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:02.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8324008870 2021-03-09T10:34:02.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9595 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:02.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c003680 con 0x7f8324008870 2021-03-09T10:34:02.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f8324008870 2021-03-09T10:34:02.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:02.412+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c036340 con 0x7f8324008870 2021-03-09T10:34:02.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:34:02 smithi146 bash[13136]: debug 2021-03-09T10:34:02.401+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27853 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:34:07.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:34:07.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:07.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9595 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:07.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=9595 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c09a310).stop 2021-03-09T10:34:07.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:07.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=CLOSED pgs=9595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:07.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:07.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:07.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:34:07.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:07.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:34:07.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:07.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:07.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d130 con 0x7f8324006200 2021-03-09T10:34:07.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ba10 con 0x7f8338069320 2021-03-09T10:34:07.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380691c0 con 0x7f8324005e00 2021-03-09T10:34:07.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:07.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:07.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:07.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:07.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:07.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338015ba0 con 0x7f8338069320 2021-03-09T10:34:07.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9600 cs=0 l=1 rev1=1 rx=0x7f8338025ad0 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:07.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c061910 con 0x7f8338069320 2021-03-09T10:34:07.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c007ba0 con 0x7f8338069320 2021-03-09T10:34:07.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:07.412+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0a1dd0 con 0x7f8338069320 2021-03-09T10:34:07.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:34:07 smithi146 bash[13136]: debug 2021-03-09T10:34:07.405+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27877 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:34:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:34:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9600 cs=0 l=1 rev1=1 rx=0x7f8338025ad0 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9600 cs=0 l=1 rev1=1 rx=0x7f8338025ad0 tx=0x7f833c09a310).stop 2021-03-09T10:34:12.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:12.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009c90 con 0x7f8324006200 2021-03-09T10:34:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d130 con 0x7f8324008870 2021-03-09T10:34:12.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380239a0 con 0x7f8324005e00 2021-03-09T10:34:12.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:34:12.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:12.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:34:12.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:12.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:12.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:12.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:12.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:12.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:12.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014710 con 0x7f8324008870 2021-03-09T10:34:12.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.412+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004780 secure :-1 s=READY pgs=9610 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:12.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.416+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c040240 con 0x7f8324008870 2021-03-09T10:34:12.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.416+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c06bd40 con 0x7f8324008870 2021-03-09T10:34:12.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:12.420+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0a2a20 con 0x7f8324008870 2021-03-09T10:34:12.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:34:12 smithi146 bash[13136]: debug 2021-03-09T10:34:12.413+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27905 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:34:17.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:34:17.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:17.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004780 secure :-1 s=READY pgs=9610 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:17.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004780 secure :-1 s=READY pgs=9610 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c09a310).stop 2021-03-09T10:34:17.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338004780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004780 unknown :-1 s=CLOSED pgs=9610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380691c0 con 0x7f8324006200 2021-03-09T10:34:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009c90 con 0x7f8338069320 2021-03-09T10:34:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380153c0 con 0x7f8324005e00 2021-03-09T10:34:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:34:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:17.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:34:17.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:17.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.411+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:17.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.415+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:17.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.415+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.415+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.415+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.415+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338014b30 con 0x7f8338069320 2021-03-09T10:34:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.415+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9619 cs=0 l=1 rev1=1 rx=0x7f833401afa0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:17.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.419+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340079c0 con 0x7f8338069320 2021-03-09T10:34:17.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338069320 2021-03-09T10:34:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:17.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402e2d0 con 0x7f8338069320 2021-03-09T10:34:17.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:34:17 smithi146 bash[13136]: debug 2021-03-09T10:34:17.417+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27929 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:34:22.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.419+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:34:22.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.419+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:22.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.419+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9619 cs=0 l=1 rev1=1 rx=0x7f833401afa0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:22.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.419+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9619 cs=0 l=1 rev1=1 rx=0x7f833401afa0 tx=0x7f83340482a0).stop 2021-03-09T10:34:22.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:22.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.423+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:22.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.423+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:22.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.423+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:22.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.423+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338025030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:22.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d130 con 0x7f8324006200 2021-03-09T10:34:22.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324008870 2021-03-09T10:34:22.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.423+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fec0 con 0x7f8324005e00 2021-03-09T10:34:22.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.423+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:34:22.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.423+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:22.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.423+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:34:22.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.423+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:22.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.423+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338025030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:22.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.427+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:22.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.427+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:22.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.427+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:22.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.427+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:22.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.427+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ac20 con 0x7f8324008870 2021-03-09T10:34:22.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.427+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338025030 secure :-1 s=READY pgs=9630 cs=0 l=1 rev1=1 rx=0x7f833c05e250 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:22.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.427+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8324008870 2021-03-09T10:34:22.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.431+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0629a0 con 0x7f8324008870 2021-03-09T10:34:22.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:22.435+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c010a60 con 0x7f8324008870 2021-03-09T10:34:22.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:34:22 smithi146 bash[13136]: debug 2021-03-09T10:34:22.421+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27957 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:34:27.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338025030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:34:27.427 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338025030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:27.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338025030 secure :-1 s=READY pgs=9630 cs=0 l=1 rev1=1 rx=0x7f833c05e250 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:27.428 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338025030 secure :-1 s=READY pgs=9630 cs=0 l=1 rev1=1 rx=0x7f833c05e250 tx=0x7f833c09a310).stop 2021-03-09T10:34:27.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338025030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:27.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338025030 unknown :-1 s=CLOSED pgs=9630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:27.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:27.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:27.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:34:27.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:27.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:34:27.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:27.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:27.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011db0 con 0x7f8324006200 2021-03-09T10:34:27.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009e40 con 0x7f8338069320 2021-03-09T10:34:27.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fec0 con 0x7f8324005e00 2021-03-09T10:34:27.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.427+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:27.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.431+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:27.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.431+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:27.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.431+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:27.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.431+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:27.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.431+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8338069320 2021-03-09T10:34:27.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.431+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9634 cs=0 l=1 rev1=1 rx=0x7f833c049020 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:27.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.435+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338069320 2021-03-09T10:34:27.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.435+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c007830 con 0x7f8338069320 2021-03-09T10:34:27.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:27.435+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c042cd0 con 0x7f8338069320 2021-03-09T10:34:27.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:34:27 smithi146 bash[13136]: debug 2021-03-09T10:34:27.425+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 27981 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:34:32.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:34:32.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:32.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9634 cs=0 l=1 rev1=1 rx=0x7f833c049020 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:32.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9634 cs=0 l=1 rev1=1 rx=0x7f833c049020 tx=0x7f833c09a310).stop 2021-03-09T10:34:32.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:32.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:32.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:32.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:32.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:34:32.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:32.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:32.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011db0 con 0x7f8324006200 2021-03-09T10:34:32.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338075210 con 0x7f8324008870 2021-03-09T10:34:32.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:34:32.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fec0 con 0x7f8324005e00 2021-03-09T10:34:32.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:32.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.431+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:32.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.435+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:32.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.435+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:32.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.435+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:32.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.435+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:32.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.435+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8324008870 2021-03-09T10:34:32.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.435+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023bf0 secure :-1 s=READY pgs=9643 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404ee10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:32.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.435+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440799d0 con 0x7f8324008870 2021-03-09T10:34:32.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.435+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T10:34:32.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:32.439+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8324008870 2021-03-09T10:34:32.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:34:32 smithi146 bash[13136]: debug 2021-03-09T10:34:32.433+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28009 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:34:37.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.435+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338023bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:34:37.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.435+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338023bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:37.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.435+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023bf0 secure :-1 s=READY pgs=9643 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404ee10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:37.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.435+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023bf0 secure :-1 s=READY pgs=9643 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834404ee10).stop 2021-03-09T10:34:37.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.435+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338023bf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:37.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.435+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338023bf0 unknown :-1 s=CLOSED pgs=9643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:37.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.435+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:37.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.435+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:37.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.435+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:34:37.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.435+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:34:37.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.435+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:37.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.435+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:37.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.435+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:37.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.435+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011db0 con 0x7f8324006200 2021-03-09T10:34:37.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.435+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a1e0 con 0x7f8338069320 2021-03-09T10:34:37.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.435+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009e40 con 0x7f8324005e00 2021-03-09T10:34:37.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.439+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:37.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.439+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:37.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.439+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:37.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.439+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:37.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.439+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:37.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.439+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e840 con 0x7f8338069320 2021-03-09T10:34:37.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.439+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9654 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c07c040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:37.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.443+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c064000 con 0x7f8338069320 2021-03-09T10:34:37.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.447+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c064000 con 0x7f8338069320 2021-03-09T10:34:37.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:37.447+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02ecb0 con 0x7f8338069320 2021-03-09T10:34:37.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:34:37 smithi146 bash[13136]: debug 2021-03-09T10:34:37.436+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28033 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:34:42.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:34:42.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:42.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9654 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c07c040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:42.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9654 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c07c040).stop 2021-03-09T10:34:42.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:42.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:42.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380243e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:42.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:42.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:34:42.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:42.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:42.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a1e0 con 0x7f8324006200 2021-03-09T10:34:42.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380691c0 con 0x7f8324008870 2021-03-09T10:34:42.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fec0 con 0x7f8324005e00 2021-03-09T10:34:42.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:34:42.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:42.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.443+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380243e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:42.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.447+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:42.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.447+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:42.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.447+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:42.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.447+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:42.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.447+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a8c0 con 0x7f8324008870 2021-03-09T10:34:42.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.447+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380243e0 secure :-1 s=READY pgs=9664 cs=0 l=1 rev1=1 rx=0x7f8334027930 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:42.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.447+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334027250 con 0x7f8324008870 2021-03-09T10:34:42.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8324008870 2021-03-09T10:34:42.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:42.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d530 con 0x7f8324008870 2021-03-09T10:34:42.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:34:42 smithi146 bash[13136]: debug 2021-03-09T10:34:42.440+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28061 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:34:47.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380243e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:34:47.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380243e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:47.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380243e0 secure :-1 s=READY pgs=9664 cs=0 l=1 rev1=1 rx=0x7f8334027930 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:47.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380243e0 secure :-1 s=READY pgs=9664 cs=0 l=1 rev1=1 rx=0x7f8334027930 tx=0x7f83340482a0).stop 2021-03-09T10:34:47.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380243e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:47.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380243e0 unknown :-1 s=CLOSED pgs=9664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:47.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:47.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:47.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:34:47.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:47.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:34:47.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:47.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:47.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a1e0 con 0x7f8324006200 2021-03-09T10:34:47.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074820 con 0x7f8338069320 2021-03-09T10:34:47.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026560 con 0x7f8324005e00 2021-03-09T10:34:47.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.447+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:47.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.451+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:47.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.451+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:47.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.451+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:47.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.451+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:47.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.451+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8338069320 2021-03-09T10:34:47.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.451+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9670 cs=0 l=1 rev1=1 rx=0x7f833801b150 tx=0x7f833c07c040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:47.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338069320 2021-03-09T10:34:47.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c061d60 con 0x7f8338069320 2021-03-09T10:34:47.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:47.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0a23d0 con 0x7f8338069320 2021-03-09T10:34:47.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:34:47 smithi146 bash[13136]: debug 2021-03-09T10:34:47.444+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28085 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:34:52.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:34:52.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:52.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9670 cs=0 l=1 rev1=1 rx=0x7f833801b150 tx=0x7f833c07c040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:52.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9670 cs=0 l=1 rev1=1 rx=0x7f833801b150 tx=0x7f833c07c040).stop 2021-03-09T10:34:52.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:52.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:52.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:52.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:52.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:52.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324006200 2021-03-09T10:34:52.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380691c0 con 0x7f8324008870 2021-03-09T10:34:52.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074820 con 0x7f8324005e00 2021-03-09T10:34:52.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:52.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:34:52.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:52.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:34:52.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.451+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:52.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.455+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:52.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.455+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:52.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.455+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:52.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.455+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:52.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.455+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a8c0 con 0x7f8324008870 2021-03-09T10:34:52.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.455+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 secure :-1 s=READY pgs=9678 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:52.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334064ef0 con 0x7f8324008870 2021-03-09T10:34:52.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334064ef0 con 0x7f8324008870 2021-03-09T10:34:52.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:52.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401a210 con 0x7f8324008870 2021-03-09T10:34:52.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:34:52 smithi146 bash[13136]: debug 2021-03-09T10:34:52.452+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28113 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:34:57.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.455+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:34:57.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.455+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:57.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.455+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 secure :-1 s=READY pgs=9678 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.455+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 secure :-1 s=READY pgs=9678 cs=0 l=1 rev1=1 rx=0x7f833400b3b0 tx=0x7f83340482a0).stop 2021-03-09T10:34:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 unknown :-1 s=CLOSED pgs=9678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.459+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:57.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.459+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:57.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.459+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:57.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.459+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:34:57.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.459+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:34:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.459+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.459+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380267a0 con 0x7f8324006200 2021-03-09T10:34:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ccd0 con 0x7f8338069320 2021-03-09T10:34:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324005e00 2021-03-09T10:34:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8338069320 2021-03-09T10:34:57.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.463+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9686 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c07c040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:57.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa020 con 0x7f8338069320 2021-03-09T10:34:57.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0792c0 con 0x7f8338069320 2021-03-09T10:34:57.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:34:57.467+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338069320 2021-03-09T10:34:57.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:34:57 smithi146 bash[13136]: debug 2021-03-09T10:34:57.456+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28137 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:35:02.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:35:02.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:02.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9686 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c07c040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:02.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9686 cs=0 l=1 rev1=1 rx=0x7f833c0222a0 tx=0x7f833c07c040).stop 2021-03-09T10:35:02.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:02.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:02.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:02.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:02.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:35:02.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:02.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:02.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f4e0 con 0x7f8324006200 2021-03-09T10:35:02.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074820 con 0x7f8324008870 2021-03-09T10:35:02.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017bc0 con 0x7f8324005e00 2021-03-09T10:35:02.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:02.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:35:02.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:02.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:02.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:02.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:02.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:02.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.463+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338066b60 con 0x7f8324008870 2021-03-09T10:35:02.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.467+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 secure :-1 s=READY pgs=9695 cs=0 l=1 rev1=1 rx=0x7f833401cb20 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:02.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.467+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833403b290 con 0x7f8324008870 2021-03-09T10:35:02.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.467+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404e220 con 0x7f8324008870 2021-03-09T10:35:02.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:02.471+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d2c0 con 0x7f8324008870 2021-03-09T10:35:02.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:35:02 smithi146 bash[13136]: debug 2021-03-09T10:35:02.460+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28165 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:35:07.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.466+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:35:07.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.466+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:07.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.466+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 secure :-1 s=READY pgs=9695 cs=0 l=1 rev1=1 rx=0x7f833401cb20 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:07.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.466+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 secure :-1 s=READY pgs=9695 cs=0 l=1 rev1=1 rx=0x7f833401cb20 tx=0x7f83340482a0).stop 2021-03-09T10:35:07.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:07.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.466+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069700 unknown :-1 s=CLOSED pgs=9695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:07.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.466+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:07.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.466+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:07.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.466+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:07.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065240 con 0x7f8324006200 2021-03-09T10:35:07.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074820 con 0x7f8338069320 2021-03-09T10:35:07.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.466+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e960 con 0x7f8324005e00 2021-03-09T10:35:07.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.466+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:35:07.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.466+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:35:07.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.466+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:07.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.466+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:07.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.470+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:07.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.470+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:07.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.470+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:07.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.470+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:07.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.470+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:07.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.470+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8338069320 2021-03-09T10:35:07.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.470+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9704 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834404ee10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:07.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.474+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403b220 con 0x7f8338069320 2021-03-09T10:35:07.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.474+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f8338069320 2021-03-09T10:35:07.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:07.478+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402c2c0 con 0x7f8338069320 2021-03-09T10:35:07.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:35:07 smithi146 bash[13136]: debug 2021-03-09T10:35:07.468+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28189 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:35:12.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:35:12.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:12.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9704 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834404ee10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:12.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9704 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834404ee10).stop 2021-03-09T10:35:12.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:12.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:12.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:12.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:12.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:12.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017bc0 con 0x7f8324006200 2021-03-09T10:35:12.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324008870 2021-03-09T10:35:12.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380691c0 con 0x7f8324005e00 2021-03-09T10:35:12.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:35:12.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:12.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:35:12.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:12.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:12.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:12.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:12.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:12.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:12.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.474+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338066b60 con 0x7f8324008870 2021-03-09T10:35:12.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.478+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019fa0 secure :-1 s=READY pgs=9713 cs=0 l=1 rev1=1 rx=0x7f833806c4d0 tx=0x7f833c079f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:12.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.478+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8324008870 2021-03-09T10:35:12.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.478+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0353d0 con 0x7f8324008870 2021-03-09T10:35:12.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:12.482+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c042380 con 0x7f8324008870 2021-03-09T10:35:12.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:35:12 smithi146 bash[13136]: debug 2021-03-09T10:35:12.472+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28217 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:35:17.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338019fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:35:17.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338019fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:17.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019fa0 secure :-1 s=READY pgs=9713 cs=0 l=1 rev1=1 rx=0x7f833806c4d0 tx=0x7f833c079f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:17.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019fa0 secure :-1 s=READY pgs=9713 cs=0 l=1 rev1=1 rx=0x7f833806c4d0 tx=0x7f833c079f80).stop 2021-03-09T10:35:17.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338019fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:17.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019fa0 unknown :-1 s=CLOSED pgs=9713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:17.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:17.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:17.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:17.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062970 con 0x7f8324006200 2021-03-09T10:35:17.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e960 con 0x7f8338069320 2021-03-09T10:35:17.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380729d0 con 0x7f8324005e00 2021-03-09T10:35:17.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:35:17.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:17.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:35:17.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:17.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.478+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:17.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.482+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:17.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.482+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:17.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.482+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:17.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.482+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:17.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.482+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a8c0 con 0x7f8338069320 2021-03-09T10:35:17.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.482+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9722 cs=0 l=1 rev1=1 rx=0x7f833c01a800 tx=0x7f833c079f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.486+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa030 con 0x7f8338069320 2021-03-09T10:35:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.486+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079bc0 con 0x7f8338069320 2021-03-09T10:35:17.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:17.490+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c062350 con 0x7f8338069320 2021-03-09T10:35:17.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:35:17 smithi146 bash[13136]: debug 2021-03-09T10:35:17.480+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28241 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:35:22.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.482+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:35:22.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.482+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:22.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.482+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9722 cs=0 l=1 rev1=1 rx=0x7f833c01a800 tx=0x7f833c079f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:22.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.482+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9722 cs=0 l=1 rev1=1 rx=0x7f833c01a800 tx=0x7f833c079f80).stop 2021-03-09T10:35:22.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.482+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:22.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.482+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:22.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.482+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:22.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.482+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:22.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.482+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:22.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.482+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324006200 2021-03-09T10:35:22.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.482+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064950 con 0x7f8324008870 2021-03-09T10:35:22.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.482+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dd80 con 0x7f8324005e00 2021-03-09T10:35:22.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.482+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:35:22.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.482+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:22.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.482+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:35:22.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.482+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:22.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.486+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:22.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.486+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:22.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.486+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:22.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.486+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:22.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.486+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:22.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.486+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024740 con 0x7f8324008870 2021-03-09T10:35:22.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.490+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005740 secure :-1 s=READY pgs=9731 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f83440498b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:22.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.490+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344032140 con 0x7f8324008870 2021-03-09T10:35:22.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.490+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f8324008870 2021-03-09T10:35:22.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:22.494+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402c2c0 con 0x7f8324008870 2021-03-09T10:35:22.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:35:22 smithi146 bash[13136]: debug 2021-03-09T10:35:22.484+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28269 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:35:27.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.486+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:35:27.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.486+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:27.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.486+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005740 secure :-1 s=READY pgs=9731 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f83440498b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:27.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.486+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005740 secure :-1 s=READY pgs=9731 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f83440498b0).stop 2021-03-09T10:35:27.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338005740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:27.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338005740 unknown :-1 s=CLOSED pgs=9731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:27.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:27.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:27.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:27.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007290 con 0x7f8324006200 2021-03-09T10:35:27.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021cb0 con 0x7f8338069320 2021-03-09T10:35:27.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324005e00 2021-03-09T10:35:27.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:35:27.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:35:27.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:27.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:27.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:27.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:27.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:27.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:27.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:27.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8338069320 2021-03-09T10:35:27.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.490+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9738 cs=0 l=1 rev1=1 rx=0x7f8334003b30 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:27.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.494+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334003e60 con 0x7f8338069320 2021-03-09T10:35:27.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.494+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007c30 con 0x7f8338069320 2021-03-09T10:35:27.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:27.498+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334047b70 con 0x7f8338069320 2021-03-09T10:35:27.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:35:27 smithi146 bash[13136]: debug 2021-03-09T10:35:27.488+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28293 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:35:32.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:35:32.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:32.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9738 cs=0 l=1 rev1=1 rx=0x7f8334003b30 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:32.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9738 cs=0 l=1 rev1=1 rx=0x7f8334003b30 tx=0x7f83340482a0).stop 2021-03-09T10:35:32.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:32.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:32.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338063080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:32.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:32.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:32.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380691c0 con 0x7f8324006200 2021-03-09T10:35:32.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e860 con 0x7f8324008870 2021-03-09T10:35:32.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800df00 con 0x7f8324005e00 2021-03-09T10:35:32.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:35:32.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:35:32.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:32.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:32.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338063080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:32.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:32.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:32.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:32.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:32.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338066b60 con 0x7f8324008870 2021-03-09T10:35:32.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.494+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338063080 secure :-1 s=READY pgs=9747 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:32.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.498+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401a950 con 0x7f8324008870 2021-03-09T10:35:32.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.498+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401a950 con 0x7f8324008870 2021-03-09T10:35:32.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:32.502+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334056da0 con 0x7f8324008870 2021-03-09T10:35:32.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:35:32 smithi146 bash[13136]: debug 2021-03-09T10:35:32.496+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28320 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:35:37.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:35:37.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:37.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338063080 secure :-1 s=READY pgs=9747 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:37.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338063080 secure :-1 s=READY pgs=9747 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f83340482a0).stop 2021-03-09T10:35:37.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338063080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:37.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338063080 unknown :-1 s=CLOSED pgs=9747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:37.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338063c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:37.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:37.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:35:37.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:37.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:37.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dd80 con 0x7f8324006200 2021-03-09T10:35:37.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065440 con 0x7f8338069320 2021-03-09T10:35:37.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068660 con 0x7f8324005e00 2021-03-09T10:35:37.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:35:37.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:37.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.498+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338063c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:37.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.502+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:37.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.502+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:37.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.502+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:37.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.502+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:37.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.502+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a8c0 con 0x7f8338069320 2021-03-09T10:35:37.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.502+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338063c60 secure :-1 s=READY pgs=9756 cs=0 l=1 rev1=1 rx=0x7f8334056510 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:37.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.506+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334016710 con 0x7f8338069320 2021-03-09T10:35:37.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.506+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334016510 con 0x7f8338069320 2021-03-09T10:35:37.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:37.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340400b0 con 0x7f8338069320 2021-03-09T10:35:37.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:35:37 smithi146 bash[13136]: debug 2021-03-09T10:35:37.504+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28345 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:35:42.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.502+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338063c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:35:42.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.502+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338063c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:42.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.502+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338063c60 secure :-1 s=READY pgs=9756 cs=0 l=1 rev1=1 rx=0x7f8334056510 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:42.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.502+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338063c60 secure :-1 s=READY pgs=9756 cs=0 l=1 rev1=1 rx=0x7f8334056510 tx=0x7f83340482a0).stop 2021-03-09T10:35:42.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.502+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338063c60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:42.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.502+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338063c60 unknown :-1 s=CLOSED pgs=9756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:42.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.502+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:42.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.502+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:42.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.502+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:42.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.502+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:35:42.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.506+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:42.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.506+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:42.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.506+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:35:42.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.506+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:42.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.506+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380239a0 con 0x7f8324006200 2021-03-09T10:35:42.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.506+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380752b0 con 0x7f8324008870 2021-03-09T10:35:42.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.506+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324005e00 2021-03-09T10:35:42.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.506+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:42.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.506+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:42.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.506+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:42.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.506+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:42.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.506+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8324008870 2021-03-09T10:35:42.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.506+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009c90 secure :-1 s=READY pgs=9764 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f83440498b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:42.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f8324008870 2021-03-09T10:35:42.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8324008870 2021-03-09T10:35:42.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:42.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f8324008870 2021-03-09T10:35:42.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:35:42 smithi146 bash[13136]: debug 2021-03-09T10:35:42.507+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28372 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:35:47.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338009c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:35:47.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338009c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:47.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009c90 secure :-1 s=READY pgs=9764 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f83440498b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:47.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009c90 secure :-1 s=READY pgs=9764 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f83440498b0).stop 2021-03-09T10:35:47.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338009c90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:47.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009c90 unknown :-1 s=CLOSED pgs=9764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:47.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:47.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:47.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:47.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380288d0 con 0x7f8324006200 2021-03-09T10:35:47.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065440 con 0x7f8338069320 2021-03-09T10:35:47.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380095f0 con 0x7f8324005e00 2021-03-09T10:35:47.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:35:47.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:35:47.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:47.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:47.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:47.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:47.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:47.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:47.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:47.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065240 con 0x7f8338069320 2021-03-09T10:35:47.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.510+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a880 secure :-1 s=READY pgs=9773 cs=0 l=1 rev1=1 rx=0x7f833404f790 tx=0x7f8334053000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:47.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404b590 con 0x7f8338069320 2021-03-09T10:35:47.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404fe70 con 0x7f8338069320 2021-03-09T10:35:47.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:47.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8338069320 2021-03-09T10:35:48.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:35:47 smithi146 bash[13136]: debug 2021-03-09T10:35:47.511+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28398 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:35:52.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:35:52.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:52.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a880 secure :-1 s=READY pgs=9773 cs=0 l=1 rev1=1 rx=0x7f833404f790 tx=0x7f8334053000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:52.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a880 secure :-1 s=READY pgs=9773 cs=0 l=1 rev1=1 rx=0x7f833404f790 tx=0x7f8334053000).stop 2021-03-09T10:35:52.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833801a880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:52.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833801a880 unknown :-1 s=CLOSED pgs=9773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:52.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:52.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:52.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:52.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:35:52.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:52.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:35:52.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:52.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cf70 con 0x7f8324006200 2021-03-09T10:35:52.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324008870 2021-03-09T10:35:52.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380752b0 con 0x7f8324005e00 2021-03-09T10:35:52.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:52.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:52.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:52.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:52.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:52.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024740 con 0x7f8324008870 2021-03-09T10:35:52.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.514+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019d40 secure :-1 s=READY pgs=9782 cs=0 l=1 rev1=1 rx=0x7f8334071fe0 tx=0x7f8334053000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:52.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833405c630 con 0x7f8324008870 2021-03-09T10:35:52.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833405c630 con 0x7f8324008870 2021-03-09T10:35:52.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:52.522+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404fb30 con 0x7f8324008870 2021-03-09T10:35:53.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:35:52 smithi146 bash[13136]: debug 2021-03-09T10:35:52.515+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28424 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:35:57.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338019d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:35:57.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338019d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:57.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019d40 secure :-1 s=READY pgs=9782 cs=0 l=1 rev1=1 rx=0x7f8334071fe0 tx=0x7f8334053000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:57.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019d40 secure :-1 s=READY pgs=9782 cs=0 l=1 rev1=1 rx=0x7f8334071fe0 tx=0x7f8334053000).stop 2021-03-09T10:35:57.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338019d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:57.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019d40 unknown :-1 s=CLOSED pgs=9782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:57.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:57.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:57.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:57.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dd80 con 0x7f8324006200 2021-03-09T10:35:57.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8338069320 2021-03-09T10:35:57.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380623d0 con 0x7f8324005e00 2021-03-09T10:35:57.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:35:57.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:57.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:35:57.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:57.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.518+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:57.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.522+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:57.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.522+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:57.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.522+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:57.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.522+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:57.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.522+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012670 con 0x7f8338069320 2021-03-09T10:35:57.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.522+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800bcc0 secure :-1 s=READY pgs=9790 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f8334015f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:57.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.522+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334019de0 con 0x7f8338069320 2021-03-09T10:35:57.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.522+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334019de0 con 0x7f8338069320 2021-03-09T10:35:57.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:35:57.526+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404fd40 con 0x7f8338069320 2021-03-09T10:35:58.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:35:57 smithi146 bash[13136]: debug 2021-03-09T10:35:57.519+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28450 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:36:02.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:36:02.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:02.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800bcc0 secure :-1 s=READY pgs=9790 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f8334015f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:02.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800bcc0 secure :-1 s=READY pgs=9790 cs=0 l=1 rev1=1 rx=0x7f83340284a0 tx=0x7f8334015f80).stop 2021-03-09T10:36:02.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833800bcc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:02.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833800bcc0 unknown :-1 s=CLOSED pgs=9790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:02.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:02.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:02.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:36:02.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:02.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:02.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:36:02.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:02.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065440 con 0x7f8324006200 2021-03-09T10:36:02.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380752b0 con 0x7f8324008870 2021-03-09T10:36:02.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ca10 con 0x7f8324005e00 2021-03-09T10:36:02.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:02.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:02.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:02.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:02.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.521+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:02.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.525+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8324008870 2021-03-09T10:36:02.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.525+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f230 secure :-1 s=READY pgs=9799 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f83440498b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:02.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f8324008870 2021-03-09T10:36:02.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T10:36:02.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:02.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344010d00 con 0x7f8324008870 2021-03-09T10:36:03.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:36:02 smithi146 bash[13136]: debug 2021-03-09T10:36:02.523+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28475 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:36:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.525+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:36:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.525+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.525+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f230 secure :-1 s=READY pgs=9799 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f83440498b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.525+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f230 secure :-1 s=READY pgs=9799 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f83440498b0).stop 2021-03-09T10:36:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.525+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801f230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f230 unknown :-1 s=CLOSED pgs=9799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.525+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.525+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:36:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.525+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.525+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.525+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:36:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.529+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f8324006200 2021-03-09T10:36:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064020 con 0x7f8338069320 2021-03-09T10:36:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.529+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065440 con 0x7f8324005e00 2021-03-09T10:36:07.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.529+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:07.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.529+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:07.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.529+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:07.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.529+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:07.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.529+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338069170 con 0x7f8338069320 2021-03-09T10:36:07.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.529+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9809 cs=0 l=1 rev1=1 rx=0x7f83340277f0 tx=0x7f8334015f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:07.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404b860 con 0x7f8338069320 2021-03-09T10:36:07.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833400dcf0 con 0x7f8338069320 2021-03-09T10:36:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:07.537+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334070b40 con 0x7f8338069320 2021-03-09T10:36:08.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:36:07 smithi146 bash[13136]: debug 2021-03-09T10:36:07.531+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28502 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:36:12.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:36:12.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:12.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9809 cs=0 l=1 rev1=1 rx=0x7f83340277f0 tx=0x7f8334015f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:12.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9809 cs=0 l=1 rev1=1 rx=0x7f83340277f0 tx=0x7f8334015f80).stop 2021-03-09T10:36:12.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:12.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:12.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:12.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:12.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:12.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024c50 con 0x7f8324006200 2021-03-09T10:36:12.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380752b0 con 0x7f8324008870 2021-03-09T10:36:12.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380148f0 con 0x7f8324005e00 2021-03-09T10:36:12.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:36:12.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:12.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:12.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:36:12.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:12.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:12.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:12.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:12.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:12.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380635a0 con 0x7f8324008870 2021-03-09T10:36:12.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.533+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073820 secure :-1 s=READY pgs=9816 cs=0 l=1 rev1=1 rx=0x7f833c093570 tx=0x7f833c079f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:12.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.537+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8324008870 2021-03-09T10:36:12.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.537+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01b700 con 0x7f8324008870 2021-03-09T10:36:12.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:12.541+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00f300 con 0x7f8324008870 2021-03-09T10:36:13.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:36:12 smithi146 bash[13136]: debug 2021-03-09T10:36:12.535+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28527 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:36:17.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338073820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:36:17.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338073820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:17.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073820 secure :-1 s=READY pgs=9816 cs=0 l=1 rev1=1 rx=0x7f833c093570 tx=0x7f833c079f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:17.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073820 secure :-1 s=READY pgs=9816 cs=0 l=1 rev1=1 rx=0x7f833c093570 tx=0x7f833c079f80).stop 2021-03-09T10:36:17.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338073820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:17.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073820 unknown :-1 s=CLOSED pgs=9816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:17.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:17.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:17.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:17.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:36:17.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:17.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063460 con 0x7f8324006200 2021-03-09T10:36:17.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dd80 con 0x7f8338069320 2021-03-09T10:36:17.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062970 con 0x7f8324005e00 2021-03-09T10:36:17.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:36:17.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:17.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:17.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:17.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:17.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:17.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.537+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:17.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.541+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338015570 con 0x7f8338069320 2021-03-09T10:36:17.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.541+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9825 cs=0 l=1 rev1=1 rx=0x7f833c026950 tx=0x7f833c079f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:17.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.541+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c026380 con 0x7f8338069320 2021-03-09T10:36:17.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.541+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0264e0 con 0x7f8338069320 2021-03-09T10:36:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:17.545+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c026380 con 0x7f8338069320 2021-03-09T10:36:18.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:36:17 smithi146 bash[13136]: debug 2021-03-09T10:36:17.539+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28554 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:36:22.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:36:22.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:22.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9825 cs=0 l=1 rev1=1 rx=0x7f833c026950 tx=0x7f833c079f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:22.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9825 cs=0 l=1 rev1=1 rx=0x7f833c026950 tx=0x7f833c079f80).stop 2021-03-09T10:36:22.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:22.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:22.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:22.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:22.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:22.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324006200 2021-03-09T10:36:22.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f8324008870 2021-03-09T10:36:22.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ec20 con 0x7f8324005e00 2021-03-09T10:36:22.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:36:22.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:22.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:22.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:36:22.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.541+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:22.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.545+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:22.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.545+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:22.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.545+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:22.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.545+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:22.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.545+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8324008870 2021-03-09T10:36:22.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.545+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009960 secure :-1 s=READY pgs=9834 cs=0 l=1 rev1=1 rx=0x7f833402dcb0 tx=0x7f8334015f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:22.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.549+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334000c90 con 0x7f8324008870 2021-03-09T10:36:22.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.549+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007c30 con 0x7f8324008870 2021-03-09T10:36:22.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:22.553+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340077b0 con 0x7f8324008870 2021-03-09T10:36:23.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:36:22 smithi146 bash[13136]: debug 2021-03-09T10:36:22.547+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28579 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:36:27.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338009960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:36:27.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338009960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:27.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009960 secure :-1 s=READY pgs=9834 cs=0 l=1 rev1=1 rx=0x7f833402dcb0 tx=0x7f8334015f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:27.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009960 secure :-1 s=READY pgs=9834 cs=0 l=1 rev1=1 rx=0x7f833402dcb0 tx=0x7f8334015f80).stop 2021-03-09T10:36:27.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338009960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:27.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338009960 unknown :-1 s=CLOSED pgs=9834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:27.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:27.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:27.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:27.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062970 con 0x7f8324006200 2021-03-09T10:36:27.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023f70 con 0x7f8338069320 2021-03-09T10:36:27.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f8324005e00 2021-03-09T10:36:27.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:36:27.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:27.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:36:27.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:27.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:27.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:27.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:27.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:27.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:27.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068400 con 0x7f8338069320 2021-03-09T10:36:27.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.549+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9842 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f83440498b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:27.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.557+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8338069320 2021-03-09T10:36:27.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.561+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:36:27.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:27.561+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834403bc90 con 0x7f8338069320 2021-03-09T10:36:28.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:36:27 smithi146 bash[13136]: debug 2021-03-09T10:36:27.551+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28606 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:36:32.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:36:32.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:32.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9842 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f83440498b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:32.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9842 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f83440498b0).stop 2021-03-09T10:36:32.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:32.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:32.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:32.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:32.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:32.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007780 con 0x7f8324006200 2021-03-09T10:36:32.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023f70 con 0x7f8324008870 2021-03-09T10:36:32.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f6a0 con 0x7f8324005e00 2021-03-09T10:36:32.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:36:32.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:32.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:36:32.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:32.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:32.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:32.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:32.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:32.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.553+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:32.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.557+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801a220 con 0x7f8324008870 2021-03-09T10:36:32.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.557+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802cd20 secure :-1 s=READY pgs=9851 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f8334015f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:32.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.557+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404b1a0 con 0x7f8324008870 2021-03-09T10:36:32.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.557+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403a400 con 0x7f8324008870 2021-03-09T10:36:32.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:32.561+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404fb30 con 0x7f8324008870 2021-03-09T10:36:33.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:36:32 smithi146 bash[13136]: debug 2021-03-09T10:36:32.555+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28631 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:36:37.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833802cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:36:37.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833802cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:37.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802cd20 secure :-1 s=READY pgs=9851 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f8334015f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:37.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802cd20 secure :-1 s=READY pgs=9851 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f8334015f80).stop 2021-03-09T10:36:37.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833802cd20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:37.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802cd20 unknown :-1 s=CLOSED pgs=9851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:37.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:37.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:37.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:37.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a390 con 0x7f8324006200 2021-03-09T10:36:37.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003060 con 0x7f8338069320 2021-03-09T10:36:37.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380288d0 con 0x7f8324005e00 2021-03-09T10:36:37.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:36:37.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:37.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:36:37.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:37.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.557+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:37.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.561+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:37.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.561+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:37.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.561+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:37.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.561+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:37.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.561+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065440 con 0x7f8338069320 2021-03-09T10:36:37.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.561+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002ab0 secure :-1 s=READY pgs=9862 cs=0 l=1 rev1=1 rx=0x7f833c01d040 tx=0x7f833c079f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:37.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.561+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338069320 2021-03-09T10:36:37.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.561+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f8338069320 2021-03-09T10:36:37.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:37.565+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338069320 2021-03-09T10:36:38.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:36:37 smithi146 bash[13136]: debug 2021-03-09T10:36:37.559+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28657 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:36:42.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338002ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:36:42.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338002ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:42.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002ab0 secure :-1 s=READY pgs=9862 cs=0 l=1 rev1=1 rx=0x7f833c01d040 tx=0x7f833c079f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:42.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002ab0 secure :-1 s=READY pgs=9862 cs=0 l=1 rev1=1 rx=0x7f833c01d040 tx=0x7f833c079f80).stop 2021-03-09T10:36:42.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338002ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:42.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338002ab0 unknown :-1 s=CLOSED pgs=9862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:42.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:42.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:42.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:42.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380749d0 con 0x7f8324006200 2021-03-09T10:36:42.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014250 con 0x7f8324008870 2021-03-09T10:36:42.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380171b0 con 0x7f8324005e00 2021-03-09T10:36:42.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:36:42.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:42.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:36:42.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:42.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.561+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:42.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.565+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:42.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.565+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:42.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.565+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:42.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.565+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:42.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.565+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012830 con 0x7f8324008870 2021-03-09T10:36:42.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.565+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802cd20 secure :-1 s=READY pgs=9870 cs=0 l=1 rev1=1 rx=0x7f833401ac50 tx=0x7f8334015f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:42.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.569+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401acc0 con 0x7f8324008870 2021-03-09T10:36:42.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.573+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401acc0 con 0x7f8324008870 2021-03-09T10:36:42.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:42.573+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404b1a0 con 0x7f8324008870 2021-03-09T10:36:43.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:36:42 smithi146 bash[13136]: debug 2021-03-09T10:36:42.566+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28682 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:36:47.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833802cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:36:47.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833802cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802cd20 secure :-1 s=READY pgs=9870 cs=0 l=1 rev1=1 rx=0x7f833401ac50 tx=0x7f8334015f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802cd20 secure :-1 s=READY pgs=9870 cs=0 l=1 rev1=1 rx=0x7f833401ac50 tx=0x7f8334015f80).stop 2021-03-09T10:36:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833802cd20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802cd20 unknown :-1 s=CLOSED pgs=9870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:47.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:47.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:47.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064020 con 0x7f8324006200 2021-03-09T10:36:47.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065860 con 0x7f8338069320 2021-03-09T10:36:47.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800da90 con 0x7f8324005e00 2021-03-09T10:36:47.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:36:47.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:47.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:36:47.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:47.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:47.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:47.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:47.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:47.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:47.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a1f0 con 0x7f8338069320 2021-03-09T10:36:47.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.569+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9877 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f833403a500).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:47.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.573+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334027a60 con 0x7f8338069320 2021-03-09T10:36:47.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.573+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334031e90 con 0x7f8338069320 2021-03-09T10:36:47.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:47.577+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401a210 con 0x7f8338069320 2021-03-09T10:36:48.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:36:47 smithi146 bash[13136]: debug 2021-03-09T10:36:47.570+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28709 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:36:52.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:36:52.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:52.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9877 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f833403a500).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:52.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9877 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f833403a500).stop 2021-03-09T10:36:52.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:52.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:52.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:52.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:52.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:52.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063460 con 0x7f8324006200 2021-03-09T10:36:52.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c940 con 0x7f8324008870 2021-03-09T10:36:52.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004bc0 con 0x7f8324005e00 2021-03-09T10:36:52.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:36:52.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:52.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:36:52.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:52.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:52.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:52.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:52.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:52.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:52.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.572+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b160 con 0x7f8324008870 2021-03-09T10:36:52.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.576+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 secure :-1 s=READY pgs=9887 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f83440498b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:52.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.576+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405f310 con 0x7f8324008870 2021-03-09T10:36:52.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.576+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T10:36:52.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:52.580+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f8324008870 2021-03-09T10:36:53.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:36:52 smithi146 bash[13136]: debug 2021-03-09T10:36:52.574+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28734 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:36:57.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:36:57.578 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:57.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 secure :-1 s=READY pgs=9887 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f83440498b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:57.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 secure :-1 s=READY pgs=9887 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f83440498b0).stop 2021-03-09T10:36:57.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:57.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 unknown :-1 s=CLOSED pgs=9887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:57.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:57.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:36:57.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:57.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:57.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:57.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004bc0 con 0x7f8324006200 2021-03-09T10:36:57.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dd80 con 0x7f8338069320 2021-03-09T10:36:57.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324005e00 2021-03-09T10:36:57.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:57.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:36:57.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.576+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:57.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.580+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:57.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.580+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:57.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.580+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:57.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.580+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:57.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.580+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024780 con 0x7f8338069320 2021-03-09T10:36:57.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.580+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9896 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f83440498b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:57.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.580+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f8338069320 2021-03-09T10:36:57.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.584+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344017640 con 0x7f8338069320 2021-03-09T10:36:57.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:36:57.588+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ac90 con 0x7f8338069320 2021-03-09T10:36:58.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:36:57 smithi146 bash[13136]: debug 2021-03-09T10:36:57.578+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28762 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:37:02.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:37:02.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:02.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9896 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f83440498b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:02.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9896 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f83440498b0).stop 2021-03-09T10:37:02.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:02.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:02.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:02.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:02.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:37:02.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:02.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:02.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c940 con 0x7f8324006200 2021-03-09T10:37:02.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024640 con 0x7f8324008870 2021-03-09T10:37:02.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004bc0 con 0x7f8324005e00 2021-03-09T10:37:02.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:37:02.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:02.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:02.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:02.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:02.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:02.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:02.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380635a0 con 0x7f8324008870 2021-03-09T10:37:02.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 secure :-1 s=READY pgs=9903 cs=0 l=1 rev1=1 rx=0x7f833c042540 tx=0x7f833c079f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:02.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.584+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa040 con 0x7f8324008870 2021-03-09T10:37:02.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.588+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00f300 con 0x7f8324008870 2021-03-09T10:37:02.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:02.588+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0040e0 con 0x7f8324008870 2021-03-09T10:37:03.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:37:02 smithi146 bash[13136]: debug 2021-03-09T10:37:02.586+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28786 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:37:07.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.584+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:37:07.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.584+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:07.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.584+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 secure :-1 s=READY pgs=9903 cs=0 l=1 rev1=1 rx=0x7f833c042540 tx=0x7f833c079f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:07.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.584+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 secure :-1 s=READY pgs=9903 cs=0 l=1 rev1=1 rx=0x7f833c042540 tx=0x7f833c079f80).stop 2021-03-09T10:37:07.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.584+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:07.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.584+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 unknown :-1 s=CLOSED pgs=9903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:07.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.584+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:07.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.584+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:07.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.584+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:07.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.584+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b930 con 0x7f8324006200 2021-03-09T10:37:07.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.584+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c940 con 0x7f8338069320 2021-03-09T10:37:07.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.584+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380629a0 con 0x7f8324005e00 2021-03-09T10:37:07.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.584+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:37:07.592 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.584+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:07.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.584+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:37:07.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.588+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:07.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.588+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:07.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.588+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:07.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.588+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:07.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.588+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:07.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.588+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:07.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.588+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338015af0 con 0x7f8338069320 2021-03-09T10:37:07.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.588+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9912 cs=0 l=1 rev1=1 rx=0x7f8338028e90 tx=0x7f834403bf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:07.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.592+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:37:07.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.592+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8338069320 2021-03-09T10:37:07.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:07.596+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402c2c0 con 0x7f8338069320 2021-03-09T10:37:08.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:37:07 smithi146 bash[13136]: debug 2021-03-09T10:37:07.590+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28814 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:37:12.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:37:12.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:12.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9912 cs=0 l=1 rev1=1 rx=0x7f8338028e90 tx=0x7f834403bf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:12.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9912 cs=0 l=1 rev1=1 rx=0x7f8338028e90 tx=0x7f834403bf90).stop 2021-03-09T10:37:12.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062580 con 0x7f8324006200 2021-03-09T10:37:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380055c0 con 0x7f8324008870 2021-03-09T10:37:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065030 con 0x7f8324005e00 2021-03-09T10:37:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:37:12.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:37:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065440 con 0x7f8324008870 2021-03-09T10:37:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.592+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 secure :-1 s=READY pgs=9920 cs=0 l=1 rev1=1 rx=0x7f8344007ce0 tx=0x7f834403bf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:12.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.596+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344012560 con 0x7f8324008870 2021-03-09T10:37:12.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.596+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8324008870 2021-03-09T10:37:12.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:12.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ac90 con 0x7f8324008870 2021-03-09T10:37:13.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:37:12 smithi146 bash[13136]: debug 2021-03-09T10:37:12.594+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28838 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:37:17.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:37:17.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:17.599 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 secure :-1 s=READY pgs=9920 cs=0 l=1 rev1=1 rx=0x7f8344007ce0 tx=0x7f834403bf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:17.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 secure :-1 s=READY pgs=9920 cs=0 l=1 rev1=1 rx=0x7f8344007ce0 tx=0x7f834403bf90).stop 2021-03-09T10:37:17.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:17.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 unknown :-1 s=CLOSED pgs=9920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:17.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:17.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:17.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:17.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:37:17.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:17.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:37:17.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:17.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324006200 2021-03-09T10:37:17.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f530 con 0x7f8338069320 2021-03-09T10:37:17.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062580 con 0x7f8324005e00 2021-03-09T10:37:17.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:17.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:17.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:17.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:17.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:17.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022220 con 0x7f8338069320 2021-03-09T10:37:17.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9929 cs=0 l=1 rev1=1 rx=0x7f834403d040 tx=0x7f834403bf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:17.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.600+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:37:17.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033030 con 0x7f8338069320 2021-03-09T10:37:17.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:17.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405d1d0 con 0x7f8338069320 2021-03-09T10:37:18.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:37:17 smithi146 bash[13136]: debug 2021-03-09T10:37:17.598+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28865 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:37:22.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:37:22.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:22.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9929 cs=0 l=1 rev1=1 rx=0x7f834403d040 tx=0x7f834403bf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:22.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9929 cs=0 l=1 rev1=1 rx=0x7f834403d040 tx=0x7f834403bf90).stop 2021-03-09T10:37:22.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:22.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:22.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:22.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:22.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:37:22.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:22.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:22.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d530 con 0x7f8324006200 2021-03-09T10:37:22.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062580 con 0x7f8324008870 2021-03-09T10:37:22.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bbf0 con 0x7f8324005e00 2021-03-09T10:37:22.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:37:22.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:22.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:22.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:22.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:22.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:22.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:22.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338069170 con 0x7f8324008870 2021-03-09T10:37:22.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.604+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 secure :-1 s=READY pgs=9939 cs=0 l=1 rev1=1 rx=0x7f8344007ce0 tx=0x7f834403bf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:22.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344016120 con 0x7f8324008870 2021-03-09T10:37:22.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T10:37:22.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:22.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405bb60 con 0x7f8324008870 2021-03-09T10:37:23.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:37:22 smithi146 bash[13136]: debug 2021-03-09T10:37:22.606+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28890 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:37:27.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:37:27.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:27.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 secure :-1 s=READY pgs=9939 cs=0 l=1 rev1=1 rx=0x7f8344007ce0 tx=0x7f834403bf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:27.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 secure :-1 s=READY pgs=9939 cs=0 l=1 rev1=1 rx=0x7f8344007ce0 tx=0x7f834403bf90).stop 2021-03-09T10:37:27.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:27.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 unknown :-1 s=CLOSED pgs=9939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:27.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:27.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:37:27.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:27.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:27.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:27.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:37:27.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:27.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:27.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018db0 con 0x7f8324006200 2021-03-09T10:37:27.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062580 con 0x7f8338069320 2021-03-09T10:37:27.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027600 con 0x7f8324005e00 2021-03-09T10:37:27.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:27.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:27.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:27.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:27.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338003010 con 0x7f8338069320 2021-03-09T10:37:27.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.608+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9946 cs=0 l=1 rev1=1 rx=0x7f834404c690 tx=0x7f834403bf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:27.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344016120 con 0x7f8338069320 2021-03-09T10:37:27.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:37:27.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:27.616+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405bb60 con 0x7f8338069320 2021-03-09T10:37:28.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:37:27 smithi146 bash[13136]: debug 2021-03-09T10:37:27.610+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28916 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:37:32.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:37:32.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:32.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9946 cs=0 l=1 rev1=1 rx=0x7f834404c690 tx=0x7f834403bf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:32.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9946 cs=0 l=1 rev1=1 rx=0x7f834404c690 tx=0x7f834403bf90).stop 2021-03-09T10:37:32.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:32.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:32.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:32.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:37:32.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:32.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:32.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:32.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:37:32.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:32.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072b70 con 0x7f8324006200 2021-03-09T10:37:32.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f530 con 0x7f8324008870 2021-03-09T10:37:32.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011080 con 0x7f8324005e00 2021-03-09T10:37:32.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:32.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:32.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:32.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:32.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:32.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007980 con 0x7f8324008870 2021-03-09T10:37:32.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.612+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802dee0 secure :-1 s=READY pgs=9955 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c079f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:32.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.616+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa030 con 0x7f8324008870 2021-03-09T10:37:32.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.616+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8324008870 2021-03-09T10:37:32.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:32.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c230 con 0x7f8324008870 2021-03-09T10:37:33.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:37:32 smithi146 bash[13136]: debug 2021-03-09T10:37:32.614+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28942 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:37:37.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833802dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:37:37.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833802dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:37.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802dee0 secure :-1 s=READY pgs=9955 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c079f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:37.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802dee0 secure :-1 s=READY pgs=9955 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c079f80).stop 2021-03-09T10:37:37.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833802dee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:37.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833802dee0 unknown :-1 s=CLOSED pgs=9955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:37.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:37.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:37.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:37.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a1b0 con 0x7f8324006200 2021-03-09T10:37:37.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338029030 con 0x7f8338069320 2021-03-09T10:37:37.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062580 con 0x7f8324005e00 2021-03-09T10:37:37.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:37:37.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:37.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:37.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:37:37.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:37.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:37.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:37.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:37.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:37.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338027600 con 0x7f8338069320 2021-03-09T10:37:37.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9964 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f833403a500).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:37.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334031130 con 0x7f8338069320 2021-03-09T10:37:37.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334031e90 con 0x7f8338069320 2021-03-09T10:37:37.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:37.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833400b980 con 0x7f8338069320 2021-03-09T10:37:38.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:37:37 smithi146 bash[13136]: debug 2021-03-09T10:37:37.618+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28968 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:37:42.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:37:42.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:42.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9964 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f833403a500).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:42.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9964 cs=0 l=1 rev1=1 rx=0x7f83340482e0 tx=0x7f833403a500).stop 2021-03-09T10:37:42.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:42.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:42.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:42.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:37:42.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:42.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:42.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:42.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:37:42.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:42.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324006200 2021-03-09T10:37:42.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bbf0 con 0x7f8324008870 2021-03-09T10:37:42.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380247d0 con 0x7f8324005e00 2021-03-09T10:37:42.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:42.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:42.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:42.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:42.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:42.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064fe0 con 0x7f8324008870 2021-03-09T10:37:42.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.623+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 secure :-1 s=READY pgs=9972 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c079f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:42.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.631+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c061510 con 0x7f8324008870 2021-03-09T10:37:42.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.631+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c061670 con 0x7f8324008870 2021-03-09T10:37:42.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:42.631+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f8324008870 2021-03-09T10:37:43.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:37:42 smithi146 bash[13136]: debug 2021-03-09T10:37:42.626+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 28994 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:37:47.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.627+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:37:47.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.627+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:47.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.627+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 secure :-1 s=READY pgs=9972 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c079f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:47.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 secure :-1 s=READY pgs=9972 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c079f80).stop 2021-03-09T10:37:47.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338007b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:47.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338007b40 unknown :-1 s=CLOSED pgs=9972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:47.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:47.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:47.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:47.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324006200 2021-03-09T10:37:47.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dca0 con 0x7f8338069320 2021-03-09T10:37:47.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011080 con 0x7f8324005e00 2021-03-09T10:37:47.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:37:47.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:47.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:47.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:37:47.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:47.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:47.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:47.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:47.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:47.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338027420 con 0x7f8338069320 2021-03-09T10:37:47.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.631+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9981 cs=0 l=1 rev1=1 rx=0x7f834403c000 tx=0x7f834403bf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:47.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.635+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:37:47.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.639+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338069320 2021-03-09T10:37:47.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:47.639+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440112f0 con 0x7f8338069320 2021-03-09T10:37:48.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:37:47 smithi146 bash[13136]: debug 2021-03-09T10:37:47.629+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29016 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:37:52.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:37:52.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:52.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9981 cs=0 l=1 rev1=1 rx=0x7f834403c000 tx=0x7f834403bf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:52.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9981 cs=0 l=1 rev1=1 rx=0x7f834403c000 tx=0x7f834403bf90).stop 2021-03-09T10:37:52.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:52.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:52.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:52.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:52.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:52.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f530 con 0x7f8324006200 2021-03-09T10:37:52.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065a60 con 0x7f8324008870 2021-03-09T10:37:52.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015a00 con 0x7f8324005e00 2021-03-09T10:37:52.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:37:52.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:52.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:37:52.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:52.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:52.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:52.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.635+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:52.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.639+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:52.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.639+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:52.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.639+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020590 con 0x7f8324008870 2021-03-09T10:37:52.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.639+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801a840 secure :-1 s=READY pgs=9991 cs=0 l=1 rev1=1 rx=0x7f833c022210 tx=0x7f833c079f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:52.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.643+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8324008870 2021-03-09T10:37:52.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.643+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c054310 con 0x7f8324008870 2021-03-09T10:37:52.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:52.643+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8324008870 2021-03-09T10:37:53.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:37:52 smithi146 bash[13136]: debug 2021-03-09T10:37:52.633+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29046 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:37:57.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.639+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:37:57.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.639+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:57.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.639+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801a840 secure :-1 s=READY pgs=9991 cs=0 l=1 rev1=1 rx=0x7f833c022210 tx=0x7f833c079f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:57.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.639+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801a840 secure :-1 s=READY pgs=9991 cs=0 l=1 rev1=1 rx=0x7f833c022210 tx=0x7f833c079f80).stop 2021-03-09T10:37:57.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801a840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:57.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801a840 unknown :-1 s=CLOSED pgs=9991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:57.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:57.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:57.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:57.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072b70 con 0x7f8324006200 2021-03-09T10:37:57.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cf30 con 0x7f8338069320 2021-03-09T10:37:57.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bbf0 con 0x7f8324005e00 2021-03-09T10:37:57.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:57.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:37:57.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:57.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:57.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:37:57.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:57.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:57.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:57.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:57.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062960 con 0x7f8338069320 2021-03-09T10:37:57.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.643+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9998 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334028410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:57.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.647+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400b980 con 0x7f8338069320 2021-03-09T10:37:57.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.647+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334007c30 con 0x7f8338069320 2021-03-09T10:37:57.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:37:57.651+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334064db0 con 0x7f8338069320 2021-03-09T10:37:58.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:37:57 smithi146 bash[13136]: debug 2021-03-09T10:37:57.641+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29071 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:38:02.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:38:02.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:02.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9998 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334028410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:02.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=9998 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334028410).stop 2021-03-09T10:38:02.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:02.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=9998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:02.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:02.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:02.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:02.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068450 con 0x7f8324006200 2021-03-09T10:38:02.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e160 con 0x7f8324008870 2021-03-09T10:38:02.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324005e00 2021-03-09T10:38:02.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:38:02.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:02.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:38:02.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:02.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.647+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:02.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.651+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:02.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.651+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:02.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.651+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:02.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.651+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:02.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.651+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380682c0 con 0x7f8324008870 2021-03-09T10:38:02.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.651+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801a840 secure :-1 s=READY pgs=10007 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334028410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:02.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.651+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334027ea0 con 0x7f8324008870 2021-03-09T10:38:02.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.651+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833400b980 con 0x7f8324008870 2021-03-09T10:38:02.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:02.655+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334064370 con 0x7f8324008870 2021-03-09T10:38:03.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:38:02 smithi146 bash[13136]: debug 2021-03-09T10:38:02.645+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29098 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:38:07.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:38:07.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:07.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801a840 secure :-1 s=READY pgs=10007 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334028410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:07.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801a840 secure :-1 s=READY pgs=10007 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334028410).stop 2021-03-09T10:38:07.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801a840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:07.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801a840 unknown :-1 s=CLOSED pgs=10007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:07.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:07.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:07.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:07.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f530 con 0x7f8324006200 2021-03-09T10:38:07.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072b70 con 0x7f8338069320 2021-03-09T10:38:07.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e7c0 con 0x7f8324005e00 2021-03-09T10:38:07.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:38:07.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:07.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:07.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:38:07.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:07.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:07.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:07.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:07.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:07.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.651+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f8338069320 2021-03-09T10:38:07.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.655+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=10016 cs=0 l=1 rev1=1 rx=0x7f833c00f580 tx=0x7f833c079f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:07.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.655+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338069320 2021-03-09T10:38:07.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.655+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f8338069320 2021-03-09T10:38:07.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:07.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c1c0 con 0x7f8338069320 2021-03-09T10:38:08.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:38:07 smithi146 bash[13136]: debug 2021-03-09T10:38:07.649+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29123 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:38:12.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.655+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:38:12.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.655+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:12.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.655+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=10016 cs=0 l=1 rev1=1 rx=0x7f833c00f580 tx=0x7f833c079f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:12.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.655+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=10016 cs=0 l=1 rev1=1 rx=0x7f833c00f580 tx=0x7f833c079f80).stop 2021-03-09T10:38:12.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.655+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:12.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.655+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=10016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:12.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.655+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338064c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:12.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.655+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:12.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.655+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:12.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.655+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801da80 con 0x7f8324006200 2021-03-09T10:38:12.659 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.655+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338013f80 con 0x7f8324008870 2021-03-09T10:38:12.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.655+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b3b0 con 0x7f8324005e00 2021-03-09T10:38:12.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.659+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338064c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:12.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.659+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:38:12.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.659+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:12.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.659+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:38:12.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.659+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:12.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.659+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:12.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.659+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:12.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.659+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:12.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.659+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:12.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.659+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b160 con 0x7f8324008870 2021-03-09T10:38:12.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.659+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338064c60 secure :-1 s=READY pgs=10024 cs=0 l=1 rev1=1 rx=0x7f8334052680 tx=0x7f8334028410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:12.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334056170 con 0x7f8324008870 2021-03-09T10:38:12.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833400b980 con 0x7f8324008870 2021-03-09T10:38:12.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:12.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401cb50 con 0x7f8324008870 2021-03-09T10:38:13.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:38:12 smithi146 bash[13136]: debug 2021-03-09T10:38:12.657+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29150 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:38:17.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.659+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338064c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:38:17.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.659+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338064c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:17.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.659+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338064c60 secure :-1 s=READY pgs=10024 cs=0 l=1 rev1=1 rx=0x7f8334052680 tx=0x7f8334028410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:17.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.659+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338064c60 secure :-1 s=READY pgs=10024 cs=0 l=1 rev1=1 rx=0x7f8334052680 tx=0x7f8334028410).stop 2021-03-09T10:38:17.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.659+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338064c60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:17.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.659+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338064c60 unknown :-1 s=CLOSED pgs=10024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:17.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.659+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:17.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.659+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:17.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.659+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.659+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e160 con 0x7f8324006200 2021-03-09T10:38:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.659+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026a80 con 0x7f8338069320 2021-03-09T10:38:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.659+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e7c0 con 0x7f8324005e00 2021-03-09T10:38:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.659+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:38:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.663+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.663+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:38:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.663+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:17.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:17.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:17.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:17.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:17.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bba0 con 0x7f8338069320 2021-03-09T10:38:17.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=10034 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c079f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:17.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338069320 2021-03-09T10:38:17.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c005db0 con 0x7f8338069320 2021-03-09T10:38:17.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:17.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c029510 con 0x7f8338069320 2021-03-09T10:38:18.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:38:17 smithi146 bash[13136]: debug 2021-03-09T10:38:17.661+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29175 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:38:22.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:38:22.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:22.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=10034 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c079f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:22.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=10034 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c079f80).stop 2021-03-09T10:38:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=10034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:22.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.667+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:22.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324006200 2021-03-09T10:38:22.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e7e0 con 0x7f8324008870 2021-03-09T10:38:22.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380026a0 con 0x7f8324005e00 2021-03-09T10:38:22.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.667+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:38:22.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.667+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:22.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.667+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:38:22.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.667+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:22.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.667+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:22.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.667+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:22.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.667+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:22.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.667+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:22.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.667+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:22.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.667+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f8324008870 2021-03-09T10:38:22.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.667+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 secure :-1 s=READY pgs=10042 cs=0 l=1 rev1=1 rx=0x7f8338031260 tx=0x7f834403bf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:22.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8324008870 2021-03-09T10:38:22.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T10:38:22.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:22.675+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440112f0 con 0x7f8324008870 2021-03-09T10:38:23.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:38:22 smithi146 bash[13136]: debug 2021-03-09T10:38:22.665+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29202 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:38:27.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:38:27.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:27.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 secure :-1 s=READY pgs=10042 cs=0 l=1 rev1=1 rx=0x7f8338031260 tx=0x7f834403bf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:27.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 secure :-1 s=READY pgs=10042 cs=0 l=1 rev1=1 rx=0x7f8338031260 tx=0x7f834403bf90).stop 2021-03-09T10:38:27.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380720f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:27.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380720f0 unknown :-1 s=CLOSED pgs=10042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:27.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:27.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:27.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:27.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380026a0 con 0x7f8324006200 2021-03-09T10:38:27.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380687c0 con 0x7f8338069320 2021-03-09T10:38:27.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028f70 con 0x7f8324005e00 2021-03-09T10:38:27.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:38:27.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:38:27.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:27.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:27.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:27.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:27.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:27.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:27.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:27.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.670+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802c300 con 0x7f8338069320 2021-03-09T10:38:27.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.674+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=10050 cs=0 l=1 rev1=1 rx=0x7f8334062970 tx=0x7f8334028410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:27.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.678+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404bd20 con 0x7f8338069320 2021-03-09T10:38:27.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334046e70 con 0x7f8338069320 2021-03-09T10:38:27.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:27.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340623c0 con 0x7f8338069320 2021-03-09T10:38:28.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:38:27 smithi146 bash[13136]: debug 2021-03-09T10:38:27.669+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29227 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:38:32.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:38:32.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:32.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=10050 cs=0 l=1 rev1=1 rx=0x7f8334062970 tx=0x7f8334028410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:32.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=10050 cs=0 l=1 rev1=1 rx=0x7f8334062970 tx=0x7f8334028410).stop 2021-03-09T10:38:32.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:32.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=10050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:32.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:32.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:32.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:32.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e7e0 con 0x7f8324006200 2021-03-09T10:38:32.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026a80 con 0x7f8324008870 2021-03-09T10:38:32.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380026a0 con 0x7f8324005e00 2021-03-09T10:38:32.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:38:32.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:32.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:38:32.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:32.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.674+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:32.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.678+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:32.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.678+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:32.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.678+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:32.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.678+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:32.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.678+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b160 con 0x7f8324008870 2021-03-09T10:38:32.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.678+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800f910 secure :-1 s=READY pgs=10059 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834403bf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:32.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405f310 con 0x7f8324008870 2021-03-09T10:38:32.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344079a60 con 0x7f8324008870 2021-03-09T10:38:32.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:32.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405bb60 con 0x7f8324008870 2021-03-09T10:38:33.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:38:32 smithi146 bash[13136]: debug 2021-03-09T10:38:32.677+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29254 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:38:37.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:38:37.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:37.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800f910 secure :-1 s=READY pgs=10059 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834403bf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:37.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800f910 secure :-1 s=READY pgs=10059 cs=0 l=1 rev1=1 rx=0x7f8344004cc0 tx=0x7f834403bf90).stop 2021-03-09T10:38:37.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800f910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:37.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800f910 unknown :-1 s=CLOSED pgs=10059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:37.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:37.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:37.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:37.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074100 con 0x7f8324006200 2021-03-09T10:38:37.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017c10 con 0x7f8338069320 2021-03-09T10:38:37.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324005e00 2021-03-09T10:38:37.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:38:37.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:37.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:38:37.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:37.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:37.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:37.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:37.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:37.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.682+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:37.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.686+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f8338069320 2021-03-09T10:38:37.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.686+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=10068 cs=0 l=1 rev1=1 rx=0x7f834403d040 tx=0x7f834403bf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:37.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400f620 con 0x7f8338069320 2021-03-09T10:38:37.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338069320 2021-03-09T10:38:37.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:37.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f8338069320 2021-03-09T10:38:38.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:38:37 smithi146 bash[13136]: debug 2021-03-09T10:38:37.681+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29279 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:38:42.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:38:42.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:42.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=10068 cs=0 l=1 rev1=1 rx=0x7f834403d040 tx=0x7f834403bf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:42.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=10068 cs=0 l=1 rev1=1 rx=0x7f834403d040 tx=0x7f834403bf90).stop 2021-03-09T10:38:42.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:42.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=10068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:42.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:42.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:42.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:42.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bbf0 con 0x7f8324006200 2021-03-09T10:38:42.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380105f0 con 0x7f8324008870 2021-03-09T10:38:42.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380026a0 con 0x7f8324005e00 2021-03-09T10:38:42.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:38:42.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:42.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:38:42.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:42.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.686+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:42.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.690+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:42.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.690+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:42.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.690+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:42.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.690+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:42.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.690+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338011b70 con 0x7f8324008870 2021-03-09T10:38:42.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.690+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 secure :-1 s=READY pgs=10077 cs=0 l=1 rev1=1 rx=0x7f8344075290 tx=0x7f834403bf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:42.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344079a60 con 0x7f8324008870 2021-03-09T10:38:42.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400f620 con 0x7f8324008870 2021-03-09T10:38:42.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:42.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834403c280 con 0x7f8324008870 2021-03-09T10:38:43.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:38:42 smithi146 bash[13136]: debug 2021-03-09T10:38:42.685+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29306 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:38:47.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:38:47.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:47.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 secure :-1 s=READY pgs=10077 cs=0 l=1 rev1=1 rx=0x7f8344075290 tx=0x7f834403bf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:47.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 secure :-1 s=READY pgs=10077 cs=0 l=1 rev1=1 rx=0x7f8344075290 tx=0x7f834403bf90).stop 2021-03-09T10:38:47.692 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:47.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338021080 unknown :-1 s=CLOSED pgs=10077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:47.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:47.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:47.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:47.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007730 con 0x7f8324006200 2021-03-09T10:38:47.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026a80 con 0x7f8338069320 2021-03-09T10:38:47.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d640 con 0x7f8324005e00 2021-03-09T10:38:47.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:38:47.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:47.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:38:47.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:47.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:47.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:47.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:47.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:47.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:47.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.690+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380083d0 con 0x7f8338069320 2021-03-09T10:38:47.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.694+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=10085 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c079f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:47.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338069320 2021-03-09T10:38:47.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338069320 2021-03-09T10:38:47.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:47.698+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004140 con 0x7f8338069320 2021-03-09T10:38:48.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:38:47 smithi146 bash[13136]: debug 2021-03-09T10:38:47.688+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29331 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:38:52.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.694+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:38:52.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.694+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:52.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.694+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=10085 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c079f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:52.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.694+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 secure :-1 s=READY pgs=10085 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c079f80).stop 2021-03-09T10:38:52.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:52.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.694+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338071450 unknown :-1 s=CLOSED pgs=10085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:52.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.694+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:52.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.694+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:52.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.694+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:52.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380026a0 con 0x7f8324006200 2021-03-09T10:38:52.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026a80 con 0x7f8324008870 2021-03-09T10:38:52.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.694+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018350 con 0x7f8324005e00 2021-03-09T10:38:52.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.694+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:38:52.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.694+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:52.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.694+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:38:52.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.694+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:52.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.698+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:52.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.698+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:52.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.698+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:52.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.698+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:52.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.698+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:52.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.698+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068770 con 0x7f8324008870 2021-03-09T10:38:52.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.698+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=10096 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c079f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:52.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01b8e0 con 0x7f8324008870 2021-03-09T10:38:52.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01ba40 con 0x7f8324008870 2021-03-09T10:38:52.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:52.706+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c010c60 con 0x7f8324008870 2021-03-09T10:38:53.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:38:52 smithi146 bash[13136]: debug 2021-03-09T10:38:52.696+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29358 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:38:57.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:38:57.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:57.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=10096 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c079f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:57.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 secure :-1 s=READY pgs=10096 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c079f80).stop 2021-03-09T10:38:57.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801ea20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:57.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801ea20 unknown :-1 s=CLOSED pgs=10096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:57.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:57.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:57.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338021d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:57.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380026a0 con 0x7f8324006200 2021-03-09T10:38:57.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d730 con 0x7f8338069320 2021-03-09T10:38:57.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066c20 con 0x7f8324005e00 2021-03-09T10:38:57.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:38:57.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:57.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:38:57.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:57.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338021d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:57.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:57.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:57.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:57.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:57.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338011030 con 0x7f8338069320 2021-03-09T10:38:57.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.702+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338021d20 secure :-1 s=READY pgs=10102 cs=0 l=1 rev1=1 rx=0x7f83380265d0 tx=0x7f833c01bbb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:57.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.706+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c005d90 con 0x7f8338069320 2021-03-09T10:38:57.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.706+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c042400 con 0x7f8338069320 2021-03-09T10:38:57.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:38:57.710+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02ec00 con 0x7f8338069320 2021-03-09T10:38:58.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:38:57 smithi146 bash[13136]: debug 2021-03-09T10:38:57.700+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29383 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:39:02.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338021d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:39:02.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338021d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:02.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338021d20 secure :-1 s=READY pgs=10102 cs=0 l=1 rev1=1 rx=0x7f83380265d0 tx=0x7f833c01bbb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:02.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338021d20 secure :-1 s=READY pgs=10102 cs=0 l=1 rev1=1 rx=0x7f83380265d0 tx=0x7f833c01bbb0).stop 2021-03-09T10:39:02.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f8338021d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:02.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f8338021d20 unknown :-1 s=CLOSED pgs=10102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:02.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:02.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:02.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:02.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e7e0 con 0x7f8324006200 2021-03-09T10:39:02.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380026a0 con 0x7f8324008870 2021-03-09T10:39:02.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066c20 con 0x7f8324005e00 2021-03-09T10:39:02.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:39:02.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:02.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:39:02.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:02.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.706+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:02.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.710+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:02.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.710+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:02.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.710+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:02.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.710+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:02.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.710+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f8324008870 2021-03-09T10:39:02.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.710+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 secure :-1 s=READY pgs=10111 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834403bf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:02.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.710+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344079a60 con 0x7f8324008870 2021-03-09T10:39:02.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.710+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T10:39:02.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:02.714+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8324008870 2021-03-09T10:39:03.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:39:02 smithi146 bash[13136]: debug 2021-03-09T10:39:02.704+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29410 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:39:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338018160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:39:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338018160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 secure :-1 s=READY pgs=10111 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834403bf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:07.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 secure :-1 s=READY pgs=10111 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834403bf90).stop 2021-03-09T10:39:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338018160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 unknown :-1 s=CLOSED pgs=10111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338061e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e7c0 con 0x7f8324006200 2021-03-09T10:39:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c940 con 0x7f8338061aa0 2021-03-09T10:39:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380026a0 con 0x7f8324005e00 2021-03-09T10:39:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:39:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:39:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:07.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338061e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:07.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:07.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338011030 con 0x7f8338061aa0 2021-03-09T10:39:07.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.714+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338061e80 secure :-1 s=READY pgs=10120 cs=0 l=1 rev1=1 rx=0x7f8334000ea0 tx=0x7f8334028410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:07.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.718+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400ba50 con 0x7f8338061aa0 2021-03-09T10:39:07.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.718+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833400ba50 con 0x7f8338061aa0 2021-03-09T10:39:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:07.722+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d290 con 0x7f8338061aa0 2021-03-09T10:39:08.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:39:07 smithi146 bash[13136]: debug 2021-03-09T10:39:07.712+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29435 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:39:12.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.713+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338061e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:39:12.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.713+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338061e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:12.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.713+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338061e80 secure :-1 s=READY pgs=10120 cs=0 l=1 rev1=1 rx=0x7f8334000ea0 tx=0x7f8334028410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:12.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.713+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338061e80 secure :-1 s=READY pgs=10120 cs=0 l=1 rev1=1 rx=0x7f8334000ea0 tx=0x7f8334028410).stop 2021-03-09T10:39:12.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338061e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:12.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338061e80 unknown :-1 s=CLOSED pgs=10120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:12.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:12.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:12.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:12.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e7e0 con 0x7f8324006200 2021-03-09T10:39:12.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380026a0 con 0x7f8324008870 2021-03-09T10:39:12.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066c20 con 0x7f8324005e00 2021-03-09T10:39:12.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:39:12.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:12.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:39:12.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:12.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:12.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:12.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:12.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:12.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:12.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a290 con 0x7f8324008870 2021-03-09T10:39:12.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.717+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 secure :-1 s=READY pgs=10128 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c01bbb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:12.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.721+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa030 con 0x7f8324008870 2021-03-09T10:39:12.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.721+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0a1690 con 0x7f8324008870 2021-03-09T10:39:12.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:12.725+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01bc20 con 0x7f8324008870 2021-03-09T10:39:13.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:39:12 smithi146 bash[13136]: debug 2021-03-09T10:39:12.716+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29462 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:39:17.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338004f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:39:17.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338004f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:17.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 secure :-1 s=READY pgs=10128 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c01bbb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:17.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 secure :-1 s=READY pgs=10128 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c01bbb0).stop 2021-03-09T10:39:17.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338004f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:17.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 unknown :-1 s=CLOSED pgs=10128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:17.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:17.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:17.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:17.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017c10 con 0x7f8324006200 2021-03-09T10:39:17.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bbf0 con 0x7f8338061aa0 2021-03-09T10:39:17.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e160 con 0x7f8324005e00 2021-03-09T10:39:17.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:39:17.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:17.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:39:17.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:17.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.721+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:17.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.725+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:17.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.725+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:17.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.725+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:17.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.725+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:17.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.725+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f8338061aa0 2021-03-09T10:39:17.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.725+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 secure :-1 s=READY pgs=10137 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f8334028410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:17.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.729+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334068620 con 0x7f8338061aa0 2021-03-09T10:39:17.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.729+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833401ae80 con 0x7f8338061aa0 2021-03-09T10:39:17.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:17.733+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401aa00 con 0x7f8338061aa0 2021-03-09T10:39:18.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:39:17 smithi146 bash[13136]: debug 2021-03-09T10:39:17.720+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29487 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:39:22.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.725+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:39:22.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.725+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:22.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.725+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 secure :-1 s=READY pgs=10137 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f8334028410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:22.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.725+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 secure :-1 s=READY pgs=10137 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f8334028410).stop 2021-03-09T10:39:22.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.725+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:22.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.725+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 unknown :-1 s=CLOSED pgs=10137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:22.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.725+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:22.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.725+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:22.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.725+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:22.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.725+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338014ca0 con 0x7f8324006200 2021-03-09T10:39:22.730 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.725+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380026a0 con 0x7f8324008870 2021-03-09T10:39:22.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.725+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cf30 con 0x7f8324005e00 2021-03-09T10:39:22.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.725+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:39:22.731 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.725+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:22.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.729+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:39:22.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.729+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:22.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.729+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:22.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.729+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:22.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.729+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:22.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.729+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:22.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.729+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:22.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.729+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380167f0 con 0x7f8324008870 2021-03-09T10:39:22.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.729+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 secure :-1 s=READY pgs=10146 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c01bbb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:22.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.733+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa040 con 0x7f8324008870 2021-03-09T10:39:22.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.733+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c042e30 con 0x7f8324008870 2021-03-09T10:39:22.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:22.737+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f8324008870 2021-03-09T10:39:23.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:39:22 smithi146 bash[13136]: debug 2021-03-09T10:39:22.728+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29513 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:39:27.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338004f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:39:27.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338004f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:27.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 secure :-1 s=READY pgs=10146 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c01bbb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:27.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 secure :-1 s=READY pgs=10146 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c01bbb0).stop 2021-03-09T10:39:27.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338004f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:27.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 unknown :-1 s=CLOSED pgs=10146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:27.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:27.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:27.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:39:27.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:27.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:27.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066110 con 0x7f8324006200 2021-03-09T10:39:27.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380026a0 con 0x7f8338061aa0 2021-03-09T10:39:27.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017c10 con 0x7f8324005e00 2021-03-09T10:39:27.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:27.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:39:27.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.733+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:27.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.737+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:27.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.737+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:27.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.737+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:27.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.737+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:27.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.737+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806afa0 con 0x7f8338061aa0 2021-03-09T10:39:27.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.737+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 secure :-1 s=READY pgs=10155 cs=0 l=1 rev1=1 rx=0x7f834403c000 tx=0x7f834403bf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:27.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.737+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8338061aa0 2021-03-09T10:39:27.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.737+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338061aa0 2021-03-09T10:39:27.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:27.741+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405dcf0 con 0x7f8338061aa0 2021-03-09T10:39:28.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:39:27 smithi146 bash[13136]: debug 2021-03-09T10:39:27.732+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29539 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:39:32.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.737+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:39:32.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.737+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:32.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.737+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 secure :-1 s=READY pgs=10155 cs=0 l=1 rev1=1 rx=0x7f834403c000 tx=0x7f834403bf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:32.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.737+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 secure :-1 s=READY pgs=10155 cs=0 l=1 rev1=1 rx=0x7f834403c000 tx=0x7f834403bf90).stop 2021-03-09T10:39:32.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.737+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:32.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.737+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 unknown :-1 s=CLOSED pgs=10155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:32.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.737+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:32.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.737+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:32.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.737+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:32.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.737+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007730 con 0x7f8324006200 2021-03-09T10:39:32.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.737+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073550 con 0x7f8324008870 2021-03-09T10:39:32.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.737+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380026a0 con 0x7f8324005e00 2021-03-09T10:39:32.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.737+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:39:32.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.737+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:32.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.737+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:39:32.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.737+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:32.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.741+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:32.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.741+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:32.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.741+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:32.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.741+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:32.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.741+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:32.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.741+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338011030 con 0x7f8324008870 2021-03-09T10:39:32.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.741+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 secure :-1 s=READY pgs=10164 cs=0 l=1 rev1=1 rx=0x7f834403d040 tx=0x7f834403bf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:32.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.745+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8324008870 2021-03-09T10:39:32.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.745+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T10:39:32.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:32.749+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344043c40 con 0x7f8324008870 2021-03-09T10:39:33.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:39:32 smithi146 bash[13136]: debug 2021-03-09T10:39:32.740+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29565 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:39:37.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338004f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:39:37.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338004f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:37.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 secure :-1 s=READY pgs=10164 cs=0 l=1 rev1=1 rx=0x7f834403d040 tx=0x7f834403bf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:37.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 secure :-1 s=READY pgs=10164 cs=0 l=1 rev1=1 rx=0x7f834403d040 tx=0x7f834403bf90).stop 2021-03-09T10:39:37.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338004f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:37.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338004f90 unknown :-1 s=CLOSED pgs=10164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:37.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:37.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:37.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:37.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e6d0 con 0x7f8324006200 2021-03-09T10:39:37.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c940 con 0x7f8338061aa0 2021-03-09T10:39:37.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062580 con 0x7f8324005e00 2021-03-09T10:39:37.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:39:37.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:37.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:37.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:39:37.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:37.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:37.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:37.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:37.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:37.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f8338061aa0 2021-03-09T10:39:37.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.745+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 secure :-1 s=READY pgs=10173 cs=0 l=1 rev1=1 rx=0x7f833c05c400 tx=0x7f833c01bbb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:37.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.749+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338061aa0 2021-03-09T10:39:37.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.749+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c036bc0 con 0x7f8338061aa0 2021-03-09T10:39:37.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:37.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338061aa0 2021-03-09T10:39:38.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:39:37 smithi146 bash[13136]: debug 2021-03-09T10:39:37.748+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29591 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:39:42.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:39:42.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:42.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 secure :-1 s=READY pgs=10173 cs=0 l=1 rev1=1 rx=0x7f833c05c400 tx=0x7f833c01bbb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:42.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 secure :-1 s=READY pgs=10173 cs=0 l=1 rev1=1 rx=0x7f833c05c400 tx=0x7f833c01bbb0).stop 2021-03-09T10:39:42.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:42.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 unknown :-1 s=CLOSED pgs=10173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:42.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:42.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:42.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:42.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:42.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338007730 con 0x7f8324006200 2021-03-09T10:39:42.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380687c0 con 0x7f8324008870 2021-03-09T10:39:42.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380729b0 con 0x7f8324005e00 2021-03-09T10:39:42.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:39:42.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:42.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:39:42.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.749+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:42.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.757+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:42.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.757+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:42.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.757+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:42.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.757+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:42.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.757+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017160 con 0x7f8324008870 2021-03-09T10:39:42.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.757+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a870 secure :-1 s=READY pgs=10183 cs=0 l=1 rev1=1 rx=0x7f8334065040 tx=0x7f8334028410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:42.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.757+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8324008870 2021-03-09T10:39:42.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.757+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8324008870 2021-03-09T10:39:42.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:42.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334064aa0 con 0x7f8324008870 2021-03-09T10:39:43.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:39:42 smithi146 bash[13136]: debug 2021-03-09T10:39:42.752+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29616 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:39:47.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:39:47.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:47.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a870 secure :-1 s=READY pgs=10183 cs=0 l=1 rev1=1 rx=0x7f8334065040 tx=0x7f8334028410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:47.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a870 secure :-1 s=READY pgs=10183 cs=0 l=1 rev1=1 rx=0x7f8334065040 tx=0x7f8334028410).stop 2021-03-09T10:39:47.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800a870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:47.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a870 unknown :-1 s=CLOSED pgs=10183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:47.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:47.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:47.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:47.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380729b0 con 0x7f8324006200 2021-03-09T10:39:47.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f8338061aa0 2021-03-09T10:39:47.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324005e00 2021-03-09T10:39:47.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:39:47.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:47.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:39:47.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:47.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:47.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:47.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:47.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:47.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:47.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.753+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065c10 con 0x7f8338061aa0 2021-03-09T10:39:47.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.757+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 secure :-1 s=READY pgs=10189 cs=0 l=1 rev1=1 rx=0x7f8338026670 tx=0x7f833c01bbb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:47.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.757+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa080 con 0x7f8338061aa0 2021-03-09T10:39:47.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c054950 con 0x7f8338061aa0 2021-03-09T10:39:47.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:47.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f8338061aa0 2021-03-09T10:39:48.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:39:47 smithi146 bash[13136]: debug 2021-03-09T10:39:47.756+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29643 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:39:52.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:39:52.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:52.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 secure :-1 s=READY pgs=10189 cs=0 l=1 rev1=1 rx=0x7f8338026670 tx=0x7f833c01bbb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:52.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 secure :-1 s=READY pgs=10189 cs=0 l=1 rev1=1 rx=0x7f8338026670 tx=0x7f833c01bbb0).stop 2021-03-09T10:39:52.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:52.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 unknown :-1 s=CLOSED pgs=10189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:52.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:52.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:52.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:52.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006200 2021-03-09T10:39:52.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d750 con 0x7f8324008870 2021-03-09T10:39:52.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008620 con 0x7f8324005e00 2021-03-09T10:39:52.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:39:52.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:52.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:39:52.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:52.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:52.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:52.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:52.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:52.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:52.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f8324008870 2021-03-09T10:39:52.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.761+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 secure :-1 s=READY pgs=10200 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834403bf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:52.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f8324008870 2021-03-09T10:39:52.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.765+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T10:39:52.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:52.769+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8324008870 2021-03-09T10:39:53.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:39:52 smithi146 bash[13136]: debug 2021-03-09T10:39:52.763+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29668 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:39:57.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.760+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338018160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:39:57.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.760+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338018160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:57.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.760+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 secure :-1 s=READY pgs=10200 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834403bf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:57.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 secure :-1 s=READY pgs=10200 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f834403bf90).stop 2021-03-09T10:39:57.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338018160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:57.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 unknown :-1 s=CLOSED pgs=10200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:57.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:57.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:39:57.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:57.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:57.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:57.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380729b0 con 0x7f8324006200 2021-03-09T10:39:57.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008620 con 0x7f8338061aa0 2021-03-09T10:39:57.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c940 con 0x7f8324005e00 2021-03-09T10:39:57.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:39:57.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:57.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:57.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:57.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.764+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.768+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806dc90 con 0x7f8338061aa0 2021-03-09T10:39:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.768+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 secure :-1 s=READY pgs=10208 cs=0 l=1 rev1=1 rx=0x7f8338066310 tx=0x7f833c01bbb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:57.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338061aa0 2021-03-09T10:39:57.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f8338061aa0 2021-03-09T10:39:57.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:39:57.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f8338061aa0 2021-03-09T10:39:58.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:39:57 smithi146 bash[13136]: debug 2021-03-09T10:39:57.767+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29695 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:40:02.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:40:02.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:02.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 secure :-1 s=READY pgs=10208 cs=0 l=1 rev1=1 rx=0x7f8338066310 tx=0x7f833c01bbb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:02.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 secure :-1 s=READY pgs=10208 cs=0 l=1 rev1=1 rx=0x7f8338066310 tx=0x7f833c01bbb0).stop 2021-03-09T10:40:02.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:02.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f8338071450 unknown :-1 s=CLOSED pgs=10208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:02.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:02.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:02.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:02.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019230 con 0x7f8324006200 2021-03-09T10:40:02.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d750 con 0x7f8324008870 2021-03-09T10:40:02.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324005e00 2021-03-09T10:40:02.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:40:02.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:02.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:02.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:40:02.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:02.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:02.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:02.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:02.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:02.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a5f0 con 0x7f8324008870 2021-03-09T10:40:02.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.768+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 secure :-1 s=READY pgs=10215 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334028410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:02.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833402d220 con 0x7f8324008870 2021-03-09T10:40:02.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.772+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402d800 con 0x7f8324008870 2021-03-09T10:40:02.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:02.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401f6f0 con 0x7f8324008870 2021-03-09T10:40:03.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:40:02 smithi146 bash[13136]: debug 2021-03-09T10:40:02.771+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29720 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:40:07.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338018160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:40:07.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338018160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:07.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 secure :-1 s=READY pgs=10215 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334028410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:07.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 secure :-1 s=READY pgs=10215 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f8334028410).stop 2021-03-09T10:40:07.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338018160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:07.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338018160 unknown :-1 s=CLOSED pgs=10215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:07.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:07.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:07.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f83380266b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:07.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380729b0 con 0x7f8324006200 2021-03-09T10:40:07.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009d30 con 0x7f8338061aa0 2021-03-09T10:40:07.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002850 con 0x7f8324005e00 2021-03-09T10:40:07.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:40:07.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:07.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:40:07.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:07.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f83380266b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:07.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:07.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:07.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:07.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:07.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f8338061aa0 2021-03-09T10:40:07.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.776+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f83380266b0 secure :-1 s=READY pgs=10224 cs=0 l=1 rev1=1 rx=0x7f833800a780 tx=0x7f833c01bbb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:07.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.780+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c036340 con 0x7f8338061aa0 2021-03-09T10:40:07.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.780+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c036980 con 0x7f8338061aa0 2021-03-09T10:40:07.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:07.784+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338061aa0 2021-03-09T10:40:08.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:40:07 smithi146 bash[13136]: debug 2021-03-09T10:40:07.779+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29747 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:40:12.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f83380266b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:40:12.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f83380266b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:12.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f83380266b0 secure :-1 s=READY pgs=10224 cs=0 l=1 rev1=1 rx=0x7f833800a780 tx=0x7f833c01bbb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:12.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f83380266b0 secure :-1 s=READY pgs=10224 cs=0 l=1 rev1=1 rx=0x7f833800a780 tx=0x7f833c01bbb0).stop 2021-03-09T10:40:12.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 msgr2=0x7f83380266b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:12.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338061aa0 0x7f83380266b0 unknown :-1 s=CLOSED pgs=10224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:12.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:12.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:12.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:12.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006200 2021-03-09T10:40:12.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008620 con 0x7f8324008870 2021-03-09T10:40:12.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002850 con 0x7f8324005e00 2021-03-09T10:40:12.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:40:12.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:12.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:40:12.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:12.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:12.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:12.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:12.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:12.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:12.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380687c0 con 0x7f8324008870 2021-03-09T10:40:12.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.780+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016a60 secure :-1 s=READY pgs=10232 cs=0 l=1 rev1=1 rx=0x7f833c00b000 tx=0x7f833c0a5040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:12.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.784+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c036550 con 0x7f8324008870 2021-03-09T10:40:12.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.784+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c007ba0 con 0x7f8324008870 2021-03-09T10:40:12.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:12.788+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c064310 con 0x7f8324008870 2021-03-09T10:40:13.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:40:12 smithi146 bash[13136]: debug 2021-03-09T10:40:12.783+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29772 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:40:17.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338016a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:40:17.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338016a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:17.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016a60 secure :-1 s=READY pgs=10232 cs=0 l=1 rev1=1 rx=0x7f833c00b000 tx=0x7f833c0a5040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:17.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016a60 secure :-1 s=READY pgs=10232 cs=0 l=1 rev1=1 rx=0x7f833c00b000 tx=0x7f833c0a5040).stop 2021-03-09T10:40:17.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338016a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:17.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016a60 unknown :-1 s=CLOSED pgs=10232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:17.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:17.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:40:17.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:17.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:17.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:17.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012460 con 0x7f8324006200 2021-03-09T10:40:17.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bbf0 con 0x7f8338016f50 2021-03-09T10:40:17.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c050 con 0x7f8324005e00 2021-03-09T10:40:17.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:40:17.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:17.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:17.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:17.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:17.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:17.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:17.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800c190 con 0x7f8338016f50 2021-03-09T10:40:17.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10241 cs=0 l=1 rev1=1 rx=0x7f8344014de0 tx=0x7f834403bf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:17.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8338016f50 2021-03-09T10:40:17.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.788+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338016f50 2021-03-09T10:40:17.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:17.792+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f8338016f50 2021-03-09T10:40:18.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:40:17 smithi146 bash[13136]: debug 2021-03-09T10:40:17.787+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29800 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:40:22.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:40:22.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:22.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10241 cs=0 l=1 rev1=1 rx=0x7f8344014de0 tx=0x7f834403bf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:22.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10241 cs=0 l=1 rev1=1 rx=0x7f8344014de0 tx=0x7f834403bf90).stop 2021-03-09T10:40:22.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:22.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=CLOSED pgs=10241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:22.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:22.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:22.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:40:22.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:40:22.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:22.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:22.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:22.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c940 con 0x7f8324006200 2021-03-09T10:40:22.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380051d0 con 0x7f8324008870 2021-03-09T10:40:22.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d750 con 0x7f8324005e00 2021-03-09T10:40:22.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:22.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:22.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:22.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:22.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:22.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802d7f0 con 0x7f8324008870 2021-03-09T10:40:22.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.792+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073820 secure :-1 s=READY pgs=10253 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c0a5040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:22.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.796+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c079a60 con 0x7f8324008870 2021-03-09T10:40:22.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.796+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c036550 con 0x7f8324008870 2021-03-09T10:40:22.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:22.800+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8324008870 2021-03-09T10:40:23.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:40:22 smithi146 bash[13136]: debug 2021-03-09T10:40:22.791+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29825 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:40:27.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338073820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:40:27.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338073820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:27.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073820 secure :-1 s=READY pgs=10253 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c0a5040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:27.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073820 secure :-1 s=READY pgs=10253 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c0a5040).stop 2021-03-09T10:40:27.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338073820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:27.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073820 unknown :-1 s=CLOSED pgs=10253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:27.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:27.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:40:27.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:27.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:27.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:27.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380729b0 con 0x7f8324006200 2021-03-09T10:40:27.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002850 con 0x7f8338016f50 2021-03-09T10:40:27.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f8324005e00 2021-03-09T10:40:27.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:40:27.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:27.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:27.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:27.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:27.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:27.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:27.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.796+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a5f0 con 0x7f8338016f50 2021-03-09T10:40:27.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.800+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10262 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834403bf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:27.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.800+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344010250 con 0x7f8338016f50 2021-03-09T10:40:27.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.800+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8338016f50 2021-03-09T10:40:27.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:27.804+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338016f50 2021-03-09T10:40:28.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:40:27 smithi146 bash[13136]: debug 2021-03-09T10:40:27.799+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29853 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:40:32.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:40:32.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:32.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10262 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834403bf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:32.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10262 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834403bf90).stop 2021-03-09T10:40:32.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:32.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=CLOSED pgs=10262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:32.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:32.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:40:32.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:32.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:32.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:32.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a3f0 con 0x7f8324006200 2021-03-09T10:40:32.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c940 con 0x7f8324008870 2021-03-09T10:40:32.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d750 con 0x7f8324005e00 2021-03-09T10:40:32.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:40:32.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:32.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.800+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:32.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.804+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:32.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.804+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:32.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.804+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:32.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.804+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:32.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.804+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380729b0 con 0x7f8324008870 2021-03-09T10:40:32.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.804+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019650 secure :-1 s=READY pgs=10269 cs=0 l=1 rev1=1 rx=0x7f833800e260 tx=0x7f833c054520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:32.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.804+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa030 con 0x7f8324008870 2021-03-09T10:40:32.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.808+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0a1320 con 0x7f8324008870 2021-03-09T10:40:32.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:32.808+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01ddc0 con 0x7f8324008870 2021-03-09T10:40:33.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:40:32 smithi146 bash[13136]: debug 2021-03-09T10:40:32.803+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29878 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:40:37.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:40:37.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:37.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019650 secure :-1 s=READY pgs=10269 cs=0 l=1 rev1=1 rx=0x7f833800e260 tx=0x7f833c054520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:37.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019650 secure :-1 s=READY pgs=10269 cs=0 l=1 rev1=1 rx=0x7f833800e260 tx=0x7f833c054520).stop 2021-03-09T10:40:37.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338019650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:37.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338019650 unknown :-1 s=CLOSED pgs=10269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:37.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:37.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:40:37.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:37.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:37.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:40:37.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:37.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f83380252d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:37.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f500 con 0x7f8324006200 2021-03-09T10:40:37.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008620 con 0x7f8338016f50 2021-03-09T10:40:37.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324005e00 2021-03-09T10:40:37.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f83380252d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:37.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:37.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:37.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:37.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:37.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e590 con 0x7f8338016f50 2021-03-09T10:40:37.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.808+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f83380252d0 secure :-1 s=READY pgs=10276 cs=0 l=1 rev1=1 rx=0x7f833c009980 tx=0x7f833c054520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:37.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.812+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa040 con 0x7f8338016f50 2021-03-09T10:40:37.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.812+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c061c90 con 0x7f8338016f50 2021-03-09T10:40:37.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:37.812+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c06b4c0 con 0x7f8338016f50 2021-03-09T10:40:38.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:40:37 smithi146 bash[13136]: debug 2021-03-09T10:40:37.807+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29906 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:40:42.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f83380252d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:40:42.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f83380252d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:42.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f83380252d0 secure :-1 s=READY pgs=10276 cs=0 l=1 rev1=1 rx=0x7f833c009980 tx=0x7f833c054520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:42.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f83380252d0 secure :-1 s=READY pgs=10276 cs=0 l=1 rev1=1 rx=0x7f833c009980 tx=0x7f833c054520).stop 2021-03-09T10:40:42.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f83380252d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:42.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f83380252d0 unknown :-1 s=CLOSED pgs=10276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:42.813 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:42.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:42.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:42.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a3f0 con 0x7f8324006200 2021-03-09T10:40:42.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d750 con 0x7f8324008870 2021-03-09T10:40:42.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a980 con 0x7f8324005e00 2021-03-09T10:40:42.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:40:42.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:42.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:40:42.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:42.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.807+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:42.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.811+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:42.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.811+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:42.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.811+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:42.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.811+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:42.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.811+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806dc90 con 0x7f8324008870 2021-03-09T10:40:42.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.811+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073f30 secure :-1 s=READY pgs=10284 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:42.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.815+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340524d0 con 0x7f8324008870 2021-03-09T10:40:42.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.815+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340646a0 con 0x7f8324008870 2021-03-09T10:40:42.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:42.819+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334064220 con 0x7f8324008870 2021-03-09T10:40:43.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:40:42 smithi146 bash[13136]: debug 2021-03-09T10:40:42.815+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29931 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:40:47.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.815+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:40:47.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.815+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:47.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.815+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073f30 secure :-1 s=READY pgs=10284 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:47.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.815+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073f30 secure :-1 s=READY pgs=10284 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f8334004c60).stop 2021-03-09T10:40:47.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.815+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338073f30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:47.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.815+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338073f30 unknown :-1 s=CLOSED pgs=10284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:47.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.815+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:47.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.819+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:47.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.819+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:40:47.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.819+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:47.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.819+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:47.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.819+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:47.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.819+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008620 con 0x7f8324006200 2021-03-09T10:40:47.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.819+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8338016f50 2021-03-09T10:40:47.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.819+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010a00 con 0x7f8324005e00 2021-03-09T10:40:47.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.819+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:40:47.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.819+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:47.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.819+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:47.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.819+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:47.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.819+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:47.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.819+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:47.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.819+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380158d0 con 0x7f8338016f50 2021-03-09T10:40:47.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.819+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10294 cs=0 l=1 rev1=1 rx=0x7f833404dd40 tx=0x7f8334004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:47.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.823+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334029b20 con 0x7f8338016f50 2021-03-09T10:40:47.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.823+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402d6a0 con 0x7f8338016f50 2021-03-09T10:40:47.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:47.827+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402e3b0 con 0x7f8338016f50 2021-03-09T10:40:48.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:40:47 smithi146 bash[13136]: debug 2021-03-09T10:40:47.819+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29959 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:40:52.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:40:52.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:52.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10294 cs=0 l=1 rev1=1 rx=0x7f833404dd40 tx=0x7f8334004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:52.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10294 cs=0 l=1 rev1=1 rx=0x7f833404dd40 tx=0x7f8334004c60).stop 2021-03-09T10:40:52.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:52.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=CLOSED pgs=10294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:52.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:52.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:52.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:52.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:40:52.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:52.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:40:52.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:52.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a3f0 con 0x7f8324006200 2021-03-09T10:40:52.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801cd10 con 0x7f8324008870 2021-03-09T10:40:52.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324005e00 2021-03-09T10:40:52.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:52.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:52.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:52.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:52.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.823+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:52.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.827+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380109b0 con 0x7f8324008870 2021-03-09T10:40:52.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.827+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 secure :-1 s=READY pgs=10302 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c054520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:52.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.831+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8324008870 2021-03-09T10:40:52.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.831+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0042e0 con 0x7f8324008870 2021-03-09T10:40:52.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:52.831+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8324008870 2021-03-09T10:40:53.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:40:52 smithi146 bash[13136]: debug 2021-03-09T10:40:52.823+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 29984 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:40:57.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.827+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338076320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:40:57.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.827+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338076320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:57.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.827+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 secure :-1 s=READY pgs=10302 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c054520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:57.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.827+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 secure :-1 s=READY pgs=10302 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c054520).stop 2021-03-09T10:40:57.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.827+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338076320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:57.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.827+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 unknown :-1 s=CLOSED pgs=10302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:57.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.827+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:57.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.827+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:57.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.827+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:57.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.827+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324006200 2021-03-09T10:40:57.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.827+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d750 con 0x7f8338016f50 2021-03-09T10:40:57.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.827+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a980 con 0x7f8324005e00 2021-03-09T10:40:57.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.827+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:40:57.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.827+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:57.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.827+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:40:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.827+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.831+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.831+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:57.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.831+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:57.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.831+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:57.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.831+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:57.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.831+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380158d0 con 0x7f8338016f50 2021-03-09T10:40:57.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.831+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10310 cs=0 l=1 rev1=1 rx=0x7f833c01e040 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:57.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.835+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c036550 con 0x7f8338016f50 2021-03-09T10:40:57.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.835+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f8338016f50 2021-03-09T10:40:57.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:40:57.839+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338016f50 2021-03-09T10:40:58.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:40:57 smithi146 bash[13136]: debug 2021-03-09T10:40:57.830+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30012 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:41:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:41:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10310 cs=0 l=1 rev1=1 rx=0x7f833c01e040 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10310 cs=0 l=1 rev1=1 rx=0x7f833c01e040 tx=0x7f833c09a310).stop 2021-03-09T10:41:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=CLOSED pgs=10310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005c00 con 0x7f8324006200 2021-03-09T10:41:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324008870 2021-03-09T10:41:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b6d0 con 0x7f8324005e00 2021-03-09T10:41:02.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:41:02.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:41:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:02.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:02.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:02.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:02.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801ff10 con 0x7f8324008870 2021-03-09T10:41:02.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.835+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 secure :-1 s=READY pgs=10319 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:02.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.839+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa040 con 0x7f8324008870 2021-03-09T10:41:02.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.839+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0a1ea0 con 0x7f8324008870 2021-03-09T10:41:02.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:02.843+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8324008870 2021-03-09T10:41:03.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:41:02 smithi146 bash[13136]: debug 2021-03-09T10:41:02.834+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30037 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:41:07.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338076320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:41:07.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338076320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:07.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 secure :-1 s=READY pgs=10319 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:07.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 secure :-1 s=READY pgs=10319 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c09a310).stop 2021-03-09T10:41:07.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338076320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:07.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 unknown :-1 s=CLOSED pgs=10319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:07.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:07.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:07.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:07.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d480 con 0x7f8324006200 2021-03-09T10:41:07.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c940 con 0x7f8338016f50 2021-03-09T10:41:07.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a980 con 0x7f8324005e00 2021-03-09T10:41:07.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:41:07.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:07.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:41:07.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:07.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.839+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:07.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.843+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:07.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.843+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:07.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.843+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:07.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.843+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:07.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.843+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802dd00 con 0x7f8338016f50 2021-03-09T10:41:07.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.843+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10329 cs=0 l=1 rev1=1 rx=0x7f833c01ba40 tx=0x7f833c09a310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:07.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.843+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa080 con 0x7f8338016f50 2021-03-09T10:41:07.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.843+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01bab0 con 0x7f8338016f50 2021-03-09T10:41:07.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:07.847+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c079b70 con 0x7f8338016f50 2021-03-09T10:41:08.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:41:07 smithi146 bash[13136]: debug 2021-03-09T10:41:07.838+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30065 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:41:12.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.843+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:41:12.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.843+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:12.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.843+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10329 cs=0 l=1 rev1=1 rx=0x7f833c01ba40 tx=0x7f833c09a310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:12.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.843+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10329 cs=0 l=1 rev1=1 rx=0x7f833c01ba40 tx=0x7f833c09a310).stop 2021-03-09T10:41:12.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.843+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:12.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.843+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=CLOSED pgs=10329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:12.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.843+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:12.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.843+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:41:12.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.843+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:12.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.843+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:12.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.843+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:12.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.843+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a3f0 con 0x7f8324006200 2021-03-09T10:41:12.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.843+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b6d0 con 0x7f8324008870 2021-03-09T10:41:12.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.843+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380689c0 con 0x7f8324005e00 2021-03-09T10:41:12.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.843+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:12.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.847+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:41:12.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.847+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:12.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.847+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:12.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.847+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:12.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.847+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:12.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.847+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:12.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.847+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338007980 con 0x7f8324008870 2021-03-09T10:41:12.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.847+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 secure :-1 s=READY pgs=10336 cs=0 l=1 rev1=1 rx=0x7f833c06bc10 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:12.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.851+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8324008870 2021-03-09T10:41:12.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.851+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c950 con 0x7f8324008870 2021-03-09T10:41:12.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:12.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01cdb0 con 0x7f8324008870 2021-03-09T10:41:13.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:41:12 smithi146 bash[13136]: debug 2021-03-09T10:41:12.846+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30090 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:41:17.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.847+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338076320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:41:17.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.847+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338076320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:17.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.847+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 secure :-1 s=READY pgs=10336 cs=0 l=1 rev1=1 rx=0x7f833c06bc10 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:17.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.847+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 secure :-1 s=READY pgs=10336 cs=0 l=1 rev1=1 rx=0x7f833c06bc10 tx=0x7f833c01b780).stop 2021-03-09T10:41:17.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.847+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338076320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:17.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.847+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 unknown :-1 s=CLOSED pgs=10336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:17.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.847+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:17.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.847+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:41:17.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.847+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:17.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.847+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:17.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.851+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:41:17.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.851+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:17.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.851+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:17.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.851+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a3f0 con 0x7f8324006200 2021-03-09T10:41:17.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.851+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e1f0 con 0x7f8338016f50 2021-03-09T10:41:17.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.851+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f8324005e00 2021-03-09T10:41:17.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.851+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:17.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.851+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:17.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.851+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:17.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.851+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:17.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.851+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:17.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.851+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380158d0 con 0x7f8338016f50 2021-03-09T10:41:17.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.851+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10345 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:17.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338016f50 2021-03-09T10:41:17.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c950 con 0x7f8338016f50 2021-03-09T10:41:17.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:17.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01cdb0 con 0x7f8338016f50 2021-03-09T10:41:18.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:41:17 smithi146 bash[13136]: debug 2021-03-09T10:41:17.850+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30117 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:41:22.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:41:22.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:22.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10345 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:22.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 secure :-1 s=READY pgs=10345 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c01b780).stop 2021-03-09T10:41:22.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338071450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:22.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338071450 unknown :-1 s=CLOSED pgs=10345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:22.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:22.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:22.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:22.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c940 con 0x7f8324006200 2021-03-09T10:41:22.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e150 con 0x7f8324008870 2021-03-09T10:41:22.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b6d0 con 0x7f8324005e00 2021-03-09T10:41:22.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:41:22.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:41:22.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:22.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:22.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:22.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:22.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:22.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:22.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:22.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806dc90 con 0x7f8324008870 2021-03-09T10:41:22.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.855+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 secure :-1 s=READY pgs=10354 cs=0 l=1 rev1=1 rx=0x7f8344048910 tx=0x7f8344041000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:22.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405bb60 con 0x7f8324008870 2021-03-09T10:41:22.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.859+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f8324008870 2021-03-09T10:41:22.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:22.863+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405bd40 con 0x7f8324008870 2021-03-09T10:41:23.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:41:22 smithi146 bash[13136]: debug 2021-03-09T10:41:22.854+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30143 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:41:27.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338076320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:41:27.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338076320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:27.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 secure :-1 s=READY pgs=10354 cs=0 l=1 rev1=1 rx=0x7f8344048910 tx=0x7f8344041000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:27.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 secure :-1 s=READY pgs=10354 cs=0 l=1 rev1=1 rx=0x7f8344048910 tx=0x7f8344041000).stop 2021-03-09T10:41:27.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338076320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:27.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338076320 unknown :-1 s=CLOSED pgs=10354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:27.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:27.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338076810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:27.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:41:27.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:27.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338076810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:27.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:27.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a3f0 con 0x7f8324006200 2021-03-09T10:41:27.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062960 con 0x7f8338016f50 2021-03-09T10:41:27.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e150 con 0x7f8324005e00 2021-03-09T10:41:27.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:41:27.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.858+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:27.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.862+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:27.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.862+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:27.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.862+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:27.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.862+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:27.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.862+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065cd0 con 0x7f8338016f50 2021-03-09T10:41:27.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.862+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338076810 secure :-1 s=READY pgs=10362 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344041000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:27.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.862+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8338016f50 2021-03-09T10:41:27.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.866+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f8338016f50 2021-03-09T10:41:27.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:27.866+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440173d0 con 0x7f8338016f50 2021-03-09T10:41:28.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:41:27 smithi146 bash[13136]: debug 2021-03-09T10:41:27.858+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30170 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:41:32.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.862+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338076810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:41:32.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.862+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338076810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:32.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.862+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338076810 secure :-1 s=READY pgs=10362 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344041000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:32.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.862+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338076810 secure :-1 s=READY pgs=10362 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344041000).stop 2021-03-09T10:41:32.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.862+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338076810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:32.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.862+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338076810 unknown :-1 s=CLOSED pgs=10362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:32.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.862+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:32.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.862+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:41:32.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.862+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:32.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.866+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:32.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.866+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:32.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.866+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e150 con 0x7f8324006200 2021-03-09T10:41:32.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.866+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324008870 2021-03-09T10:41:32.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.866+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025a30 con 0x7f8324005e00 2021-03-09T10:41:32.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.866+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:41:32.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.866+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:32.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.866+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:32.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.866+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:32.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.866+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:32.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.866+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:32.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.866+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f833802b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:32.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.866+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380158d0 con 0x7f8324008870 2021-03-09T10:41:32.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.866+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380753e0 secure :-1 s=READY pgs=10371 cs=0 l=1 rev1=1 rx=0x7f833c059040 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:32.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.870+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa030 con 0x7f8324008870 2021-03-09T10:41:32.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.870+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c026380 con 0x7f8324008870 2021-03-09T10:41:32.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:32.874+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c036340 con 0x7f8324008870 2021-03-09T10:41:33.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:41:32 smithi146 bash[13136]: debug 2021-03-09T10:41:32.866+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30196 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:41:37.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:41:37.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:37.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380753e0 secure :-1 s=READY pgs=10371 cs=0 l=1 rev1=1 rx=0x7f833c059040 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:37.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380753e0 secure :-1 s=READY pgs=10371 cs=0 l=1 rev1=1 rx=0x7f833c059040 tx=0x7f833c01b780).stop 2021-03-09T10:41:37.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380753e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:37.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380753e0 unknown :-1 s=CLOSED pgs=10371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:37.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:37.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:41:37.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:37.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:37.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:37.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062960 con 0x7f8324006200 2021-03-09T10:41:37.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a3f0 con 0x7f8338016f50 2021-03-09T10:41:37.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e150 con 0x7f8324005e00 2021-03-09T10:41:37.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:41:37.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:37.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:37.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:37.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:37.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:37.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:37.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.870+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338012200 con 0x7f8338016f50 2021-03-09T10:41:37.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.874+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802b2d0 secure :-1 s=READY pgs=10381 cs=0 l=1 rev1=1 rx=0x7f833c079480 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:37.876 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c062b00 con 0x7f8338016f50 2021-03-09T10:41:37.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c062b00 con 0x7f8338016f50 2021-03-09T10:41:37.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:37.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f8338016f50 2021-03-09T10:41:38.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:41:37 smithi146 bash[13136]: debug 2021-03-09T10:41:37.870+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30222 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:41:42.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.874+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:41:42.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.874+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:42.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.874+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802b2d0 secure :-1 s=READY pgs=10381 cs=0 l=1 rev1=1 rx=0x7f833c079480 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:42.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.874+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802b2d0 secure :-1 s=READY pgs=10381 cs=0 l=1 rev1=1 rx=0x7f833c079480 tx=0x7f833c01b780).stop 2021-03-09T10:41:42.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.874+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:42.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.874+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=10381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:42.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.874+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:42.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.874+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:42.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.874+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:42.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.874+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e150 con 0x7f8324006200 2021-03-09T10:41:42.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.874+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c940 con 0x7f8324008870 2021-03-09T10:41:42.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.874+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802cb90 con 0x7f8324005e00 2021-03-09T10:41:42.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.874+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:41:42.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.874+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:42.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.874+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:41:42.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.874+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:42.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.878+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:42.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.878+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:42.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.878+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:42.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.878+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:42.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.878+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:42.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.878+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802dd00 con 0x7f8324008870 2021-03-09T10:41:42.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.882+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074310 secure :-1 s=READY pgs=10390 cs=0 l=1 rev1=1 rx=0x7f833404d050 tx=0x7f8334004c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:42.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.882+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334031130 con 0x7f8324008870 2021-03-09T10:41:42.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.882+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8324008870 2021-03-09T10:41:42.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:42.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401ca30 con 0x7f8324008870 2021-03-09T10:41:43.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:41:42 smithi146 bash[13136]: debug 2021-03-09T10:41:42.878+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30249 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:41:47.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.878+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:41:47.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.878+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:47.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.878+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074310 secure :-1 s=READY pgs=10390 cs=0 l=1 rev1=1 rx=0x7f833404d050 tx=0x7f8334004c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:47.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.878+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074310 secure :-1 s=READY pgs=10390 cs=0 l=1 rev1=1 rx=0x7f833404d050 tx=0x7f8334004c60).stop 2021-03-09T10:41:47.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338074310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:47.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.878+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074310 unknown :-1 s=CLOSED pgs=10390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:47.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.878+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:47.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.878+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:47.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.878+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:47.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e150 con 0x7f8324006200 2021-03-09T10:41:47.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065d20 con 0x7f8338016f50 2021-03-09T10:41:47.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.878+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074d70 con 0x7f8324005e00 2021-03-09T10:41:47.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.882+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:41:47.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.882+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:47.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.882+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:41:47.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.882+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:47.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.882+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:47.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.882+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:47.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.882+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:47.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.882+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:47.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.882+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:47.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.882+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062960 con 0x7f8338016f50 2021-03-09T10:41:47.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.882+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802b2d0 secure :-1 s=READY pgs=10397 cs=0 l=1 rev1=1 rx=0x7f833403b770 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:47.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833403beb0 con 0x7f8338016f50 2021-03-09T10:41:47.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403beb0 con 0x7f8338016f50 2021-03-09T10:41:47.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:47.890+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334003350 con 0x7f8338016f50 2021-03-09T10:41:48.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:41:47 smithi146 bash[13136]: debug 2021-03-09T10:41:47.882+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30275 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:41:52.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:41:52.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:52.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802b2d0 secure :-1 s=READY pgs=10397 cs=0 l=1 rev1=1 rx=0x7f833403b770 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:52.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802b2d0 secure :-1 s=READY pgs=10397 cs=0 l=1 rev1=1 rx=0x7f833403b770 tx=0x7f83340482a0).stop 2021-03-09T10:41:52.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:52.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=10397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:52.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:52.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:52.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:52.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c940 con 0x7f8324006200 2021-03-09T10:41:52.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024bf0 con 0x7f8324008870 2021-03-09T10:41:52.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e150 con 0x7f8324005e00 2021-03-09T10:41:52.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:41:52.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:52.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:41:52.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:52.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.886+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:52.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.890+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:52.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.890+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:52.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.890+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:52.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.890+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:52.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.890+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065440 con 0x7f8324008870 2021-03-09T10:41:52.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.890+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074310 secure :-1 s=READY pgs=10406 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:52.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.894+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334031130 con 0x7f8324008870 2021-03-09T10:41:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.894+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404dbd0 con 0x7f8324008870 2021-03-09T10:41:52.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:52.898+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334003350 con 0x7f8324008870 2021-03-09T10:41:53.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:41:52 smithi146 bash[13136]: debug 2021-03-09T10:41:52.890+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30302 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:41:57.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.890+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:41:57.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.890+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:57.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.890+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074310 secure :-1 s=READY pgs=10406 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:57.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.890+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074310 secure :-1 s=READY pgs=10406 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).stop 2021-03-09T10:41:57.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.894+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338074310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:57.894 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.894+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338074310 unknown :-1 s=CLOSED pgs=10406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:57.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.894+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:57.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.894+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:57.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.894+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:57.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.894+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026270 con 0x7f8324006200 2021-03-09T10:41:57.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.894+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028620 con 0x7f8338016f50 2021-03-09T10:41:57.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.894+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380167f0 con 0x7f8324005e00 2021-03-09T10:41:57.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.894+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:41:57.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.894+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:57.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.894+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:41:57.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.894+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:57.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.894+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:57.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.898+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:57.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.898+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:57.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.898+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:57.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.898+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:57.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.898+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806dc90 con 0x7f8338016f50 2021-03-09T10:41:57.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.898+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 secure :-1 s=READY pgs=10417 cs=0 l=1 rev1=1 rx=0x7f833c05e250 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:57.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.898+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338016f50 2021-03-09T10:41:57.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.898+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f8338016f50 2021-03-09T10:41:57.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:41:57.902+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c07a310 con 0x7f8338016f50 2021-03-09T10:41:58.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:41:57 smithi146 bash[13136]: debug 2021-03-09T10:41:57.893+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30328 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:42:02.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.898+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338029c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:42:02.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.898+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338029c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:02.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.898+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 secure :-1 s=READY pgs=10417 cs=0 l=1 rev1=1 rx=0x7f833c05e250 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:02.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.898+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 secure :-1 s=READY pgs=10417 cs=0 l=1 rev1=1 rx=0x7f833c05e250 tx=0x7f833c01b780).stop 2021-03-09T10:42:02.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.898+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338029c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:02.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.898+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 unknown :-1 s=CLOSED pgs=10417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:02.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.898+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:02.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.898+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:02.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.898+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338011970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:02.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.898+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e0e0 con 0x7f8324006200 2021-03-09T10:42:02.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.898+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f6b0 con 0x7f8324008870 2021-03-09T10:42:02.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.898+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e150 con 0x7f8324005e00 2021-03-09T10:42:02.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.898+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:42:02.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.898+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:02.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.898+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338011970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:02.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.902+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:02.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.902+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:02.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.902+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:02.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.902+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:02.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.902+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802dd00 con 0x7f8324008870 2021-03-09T10:42:02.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.902+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338011970 secure :-1 s=READY pgs=10423 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:02.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.906+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c02e9a0 con 0x7f8324008870 2021-03-09T10:42:02.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.906+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f8324008870 2021-03-09T10:42:02.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:02.906+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02e9a0 con 0x7f8324008870 2021-03-09T10:42:03.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:42:02 smithi146 bash[13136]: debug 2021-03-09T10:42:02.901+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30355 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:42:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.902+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338011970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:42:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.902+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338011970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.902+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338011970 secure :-1 s=READY pgs=10423 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.902+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338011970 secure :-1 s=READY pgs=10423 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c01b780).stop 2021-03-09T10:42:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.902+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338011970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.902+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338011970 unknown :-1 s=CLOSED pgs=10423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.902+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.902+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.902+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.902+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380623c0 con 0x7f8324006200 2021-03-09T10:42:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.902+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e090 con 0x7f8338016f50 2021-03-09T10:42:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.902+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018a80 con 0x7f8324005e00 2021-03-09T10:42:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.906+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:42:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.906+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:42:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.906+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.906+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.906+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.906+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.906+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.906+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.906+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.906+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062960 con 0x7f8338016f50 2021-03-09T10:42:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.906+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 secure :-1 s=READY pgs=10433 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.910+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338016f50 2021-03-09T10:42:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.910+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0792c0 con 0x7f8338016f50 2021-03-09T10:42:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:07.914+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01c3e0 con 0x7f8338016f50 2021-03-09T10:42:08.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:42:07 smithi146 bash[13136]: debug 2021-03-09T10:42:07.905+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30381 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:42:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338029c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:42:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338029c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 secure :-1 s=READY pgs=10433 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 secure :-1 s=READY pgs=10433 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01b780).stop 2021-03-09T10:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338029c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 unknown :-1 s=CLOSED pgs=10433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e150 con 0x7f8324006200 2021-03-09T10:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068320 con 0x7f8324008870 2021-03-09T10:42:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f2e0 con 0x7f8324005e00 2021-03-09T10:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:42:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:12.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:12.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:12.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:12.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:12.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:12.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.909+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801ccc0 con 0x7f8324008870 2021-03-09T10:42:12.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.913+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069320 secure :-1 s=READY pgs=10440 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:12.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.913+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8324008870 2021-03-09T10:42:12.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.913+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334004dd0 con 0x7f8324008870 2021-03-09T10:42:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:12.917+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340191b0 con 0x7f8324008870 2021-03-09T10:42:13.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:42:12 smithi146 bash[13136]: debug 2021-03-09T10:42:12.909+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30408 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:42:17.915 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:42:17.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:17.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069320 secure :-1 s=READY pgs=10440 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:17.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069320 secure :-1 s=READY pgs=10440 cs=0 l=1 rev1=1 rx=0x7f833402ed10 tx=0x7f83340482a0).stop 2021-03-09T10:42:17.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:17.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069320 unknown :-1 s=CLOSED pgs=10440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:17.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:17.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:17.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:17.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338028620 con 0x7f8324006200 2021-03-09T10:42:17.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c940 con 0x7f8338016f50 2021-03-09T10:42:17.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380623c0 con 0x7f8324005e00 2021-03-09T10:42:17.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:42:17.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:17.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:42:17.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:17.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.913+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:17.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.917+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:17.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.917+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:17.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.917+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:17.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.917+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:17.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.917+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005010 con 0x7f8338016f50 2021-03-09T10:42:17.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.917+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 secure :-1 s=READY pgs=10451 cs=0 l=1 rev1=1 rx=0x7f8338063bc0 tx=0x7f8344041000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:17.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.917+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f8338016f50 2021-03-09T10:42:17.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.921+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338016f50 2021-03-09T10:42:17.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:17.925+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338016f50 2021-03-09T10:42:18.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:42:17 smithi146 bash[13136]: debug 2021-03-09T10:42:17.917+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30430 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:42:22.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.917+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338029c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:42:22.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.917+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338029c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:22.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.917+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 secure :-1 s=READY pgs=10451 cs=0 l=1 rev1=1 rx=0x7f8338063bc0 tx=0x7f8344041000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:22.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.917+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 secure :-1 s=READY pgs=10451 cs=0 l=1 rev1=1 rx=0x7f8338063bc0 tx=0x7f8344041000).stop 2021-03-09T10:42:22.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.917+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338029c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:22.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.917+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 unknown :-1 s=CLOSED pgs=10451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:22.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.917+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:22.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.917+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:22.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.917+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:22.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.921+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:42:22.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.921+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:22.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.921+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018a80 con 0x7f8324006200 2021-03-09T10:42:22.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.921+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e0e0 con 0x7f8324008870 2021-03-09T10:42:22.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.921+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e150 con 0x7f8324005e00 2021-03-09T10:42:22.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.921+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:42:22.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.921+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:22.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.921+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:22.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.921+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:22.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.921+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:22.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.921+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:22.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.921+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:22.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.921+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338002b10 con 0x7f8324008870 2021-03-09T10:42:22.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.921+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069320 secure :-1 s=READY pgs=10458 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:22.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.925+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833402db30 con 0x7f8324008870 2021-03-09T10:42:22.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.925+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402db30 con 0x7f8324008870 2021-03-09T10:42:22.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:22.929+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334058c30 con 0x7f8324008870 2021-03-09T10:42:23.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:42:22 smithi146 bash[13136]: debug 2021-03-09T10:42:22.921+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30456 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:42:27.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:42:27.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:27.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069320 secure :-1 s=READY pgs=10458 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:27.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069320 secure :-1 s=READY pgs=10458 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f83340482a0).stop 2021-03-09T10:42:27.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:27.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069320 unknown :-1 s=CLOSED pgs=10458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:27.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:27.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:27.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:27.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024df0 con 0x7f8324006200 2021-03-09T10:42:27.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023a20 con 0x7f8338016f50 2021-03-09T10:42:27.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066510 con 0x7f8324005e00 2021-03-09T10:42:27.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:42:27.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:27.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:42:27.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:27.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.925+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:27.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.929+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:27.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.929+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:27.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.929+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:27.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.929+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:27.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.929+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802d400 con 0x7f8338016f50 2021-03-09T10:42:27.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.929+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 secure :-1 s=READY pgs=10467 cs=0 l=1 rev1=1 rx=0x7f833404d6f0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:27.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.937+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833404d990 con 0x7f8338016f50 2021-03-09T10:42:27.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.937+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334027420 con 0x7f8338016f50 2021-03-09T10:42:27.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:27.937+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833404dc20 con 0x7f8338016f50 2021-03-09T10:42:28.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:42:27 smithi146 bash[13136]: debug 2021-03-09T10:42:27.925+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30482 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:42:32.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.929+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338029c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:42:32.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.929+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338029c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:32.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.929+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 secure :-1 s=READY pgs=10467 cs=0 l=1 rev1=1 rx=0x7f833404d6f0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:32.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.929+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 secure :-1 s=READY pgs=10467 cs=0 l=1 rev1=1 rx=0x7f833404d6f0 tx=0x7f83340482a0).stop 2021-03-09T10:42:32.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.929+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338029c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:32.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.929+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338029c00 unknown :-1 s=CLOSED pgs=10467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:32.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.929+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:32.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.929+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:32.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.929+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:32.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.929+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073140 con 0x7f8324006200 2021-03-09T10:42:32.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.929+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e150 con 0x7f8324008870 2021-03-09T10:42:32.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.929+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e2d0 con 0x7f8324005e00 2021-03-09T10:42:32.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.929+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:42:32.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.929+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:42:32.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.929+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:32.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.929+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:32.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.933+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:32.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.933+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:32.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.933+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:32.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.933+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:32.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.933+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:32.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.933+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800fe80 con 0x7f8324008870 2021-03-09T10:42:32.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.933+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069a20 secure :-1 s=READY pgs=10476 cs=0 l=1 rev1=1 rx=0x7f833404d920 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:32.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.937+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334047a00 con 0x7f8324008870 2021-03-09T10:42:32.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.937+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83340189b0 con 0x7f8324008870 2021-03-09T10:42:32.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:32.937+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83340273b0 con 0x7f8324008870 2021-03-09T10:42:33.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:42:32 smithi146 bash[13136]: debug 2021-03-09T10:42:32.933+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30509 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:42:37.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:42:37.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:37.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069a20 secure :-1 s=READY pgs=10476 cs=0 l=1 rev1=1 rx=0x7f833404d920 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:37.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069a20 secure :-1 s=READY pgs=10476 cs=0 l=1 rev1=1 rx=0x7f833404d920 tx=0x7f83340482a0).stop 2021-03-09T10:42:37.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338069a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:37.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338069a20 unknown :-1 s=CLOSED pgs=10476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:37.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:37.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:37.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:42:37.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:37.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:42:37.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833806d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:37.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:37.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011020 con 0x7f8324006200 2021-03-09T10:42:37.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380289f0 con 0x7f8338016f50 2021-03-09T10:42:37.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833806d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:37.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e0e0 con 0x7f8324005e00 2021-03-09T10:42:37.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:37.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:37.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:37.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:37.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062960 con 0x7f8338016f50 2021-03-09T10:42:37.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.937+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833806d320 secure :-1 s=READY pgs=10485 cs=0 l=1 rev1=1 rx=0x7f8338063bc0 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:37.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.941+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa040 con 0x7f8338016f50 2021-03-09T10:42:37.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.941+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0a1ef0 con 0x7f8338016f50 2021-03-09T10:42:37.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:37.945+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f8338016f50 2021-03-09T10:42:38.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:42:37 smithi146 bash[13136]: debug 2021-03-09T10:42:37.937+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30535 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:42:42.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833806d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:42:42.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833806d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:42.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833806d320 secure :-1 s=READY pgs=10485 cs=0 l=1 rev1=1 rx=0x7f8338063bc0 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:42.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833806d320 secure :-1 s=READY pgs=10485 cs=0 l=1 rev1=1 rx=0x7f8338063bc0 tx=0x7f833c01b780).stop 2021-03-09T10:42:42.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833806d320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:42.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833806d320 unknown :-1 s=CLOSED pgs=10485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:42.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380645b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:42.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:42.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:42.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073140 con 0x7f8324006200 2021-03-09T10:42:42.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023a70 con 0x7f8324008870 2021-03-09T10:42:42.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ed60 con 0x7f8324005e00 2021-03-09T10:42:42.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:42:42.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:42.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:42:42.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:42.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.941+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:42.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.945+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:42.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.945+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:42.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.945+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:42.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.945+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:42.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.945+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e590 con 0x7f8324008870 2021-03-09T10:42:42.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.945+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380645b0 secure :-1 s=READY pgs=10494 cs=0 l=1 rev1=1 rx=0x7f833404d860 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:42.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833401cc70 con 0x7f8324008870 2021-03-09T10:42:42.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.953+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833404d5f0 con 0x7f8324008870 2021-03-09T10:42:42.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:42.953+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8324008870 2021-03-09T10:42:43.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:42:42 smithi146 bash[13136]: debug 2021-03-09T10:42:42.945+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30562 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:42:47.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380645b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:42:47.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380645b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:47.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380645b0 secure :-1 s=READY pgs=10494 cs=0 l=1 rev1=1 rx=0x7f833404d860 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:47.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380645b0 secure :-1 s=READY pgs=10494 cs=0 l=1 rev1=1 rx=0x7f833404d860 tx=0x7f83340482a0).stop 2021-03-09T10:42:47.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380645b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:47.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380645b0 unknown :-1 s=CLOSED pgs=10494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:47.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:47.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:47.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:47.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066510 con 0x7f8324006200 2021-03-09T10:42:47.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cfb0 con 0x7f8338016f50 2021-03-09T10:42:47.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e0e0 con 0x7f8324005e00 2021-03-09T10:42:47.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:42:47.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:47.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:42:47.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:47.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:47.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:47.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:47.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:47.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:47.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026a40 con 0x7f8338016f50 2021-03-09T10:42:47.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.949+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833806d2a0 secure :-1 s=READY pgs=10501 cs=0 l=1 rev1=1 rx=0x7f8338005710 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:47.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.953+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338016f50 2021-03-09T10:42:47.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.953+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00f300 con 0x7f8338016f50 2021-03-09T10:42:47.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:47.957+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0040e0 con 0x7f8338016f50 2021-03-09T10:42:48.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:42:47 smithi146 bash[13136]: debug 2021-03-09T10:42:47.949+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30588 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:42:52.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:42:52.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:52.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833806d2a0 secure :-1 s=READY pgs=10501 cs=0 l=1 rev1=1 rx=0x7f8338005710 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:52.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833806d2a0 secure :-1 s=READY pgs=10501 cs=0 l=1 rev1=1 rx=0x7f8338005710 tx=0x7f833c01b780).stop 2021-03-09T10:42:52.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833806d2a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:52.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833806d2a0 unknown :-1 s=CLOSED pgs=10501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:52.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:52.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:52.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:52.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e150 con 0x7f8324006200 2021-03-09T10:42:52.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073140 con 0x7f8324008870 2021-03-09T10:42:52.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380191f0 con 0x7f8324005e00 2021-03-09T10:42:52.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:42:52.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:52.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:42:52.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:52.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:52.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:52.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:52.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:52.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.953+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:52.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.957+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338062960 con 0x7f8324008870 2021-03-09T10:42:52.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.957+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f00 secure :-1 s=READY pgs=10510 cs=0 l=1 rev1=1 rx=0x7f833c059040 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:52.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.961+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa040 con 0x7f8324008870 2021-03-09T10:42:52.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.961+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0a1320 con 0x7f8324008870 2021-03-09T10:42:52.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:52.965+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f8324008870 2021-03-09T10:42:53.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:42:52 smithi146 bash[13136]: debug 2021-03-09T10:42:52.957+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30615 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:42:57.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:42:57.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:57.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f00 secure :-1 s=READY pgs=10510 cs=0 l=1 rev1=1 rx=0x7f833c059040 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:57.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f00 secure :-1 s=READY pgs=10510 cs=0 l=1 rev1=1 rx=0x7f833c059040 tx=0x7f833c01b780).stop 2021-03-09T10:42:57.961 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338027f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:57.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338027f00 unknown :-1 s=CLOSED pgs=10510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:57.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:57.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:57.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:57.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806cfb0 con 0x7f8324006200 2021-03-09T10:42:57.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023a70 con 0x7f8338016f50 2021-03-09T10:42:57.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e150 con 0x7f8324005e00 2021-03-09T10:42:57.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:42:57.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:57.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:42:57.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:57.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:57.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:57.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:57.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:57.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:57.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005fd0 con 0x7f8338016f50 2021-03-09T10:42:57.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.960+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f160 secure :-1 s=READY pgs=10518 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344041000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:57.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.964+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8338016f50 2021-03-09T10:42:57.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.964+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338016f50 2021-03-09T10:42:57.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:42:57.968+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338016f50 2021-03-09T10:42:58.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:42:57 smithi146 bash[13136]: debug 2021-03-09T10:42:57.961+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30641 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:43:02.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833801f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:43:02.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833801f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:02.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f160 secure :-1 s=READY pgs=10518 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344041000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:02.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f160 secure :-1 s=READY pgs=10518 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344041000).stop 2021-03-09T10:43:02.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833801f160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:02.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f160 unknown :-1 s=CLOSED pgs=10518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:02.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:02.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:02.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:02.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072e50 con 0x7f8324006200 2021-03-09T10:43:02.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c300 con 0x7f8324008870 2021-03-09T10:43:02.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c540 con 0x7f8324005e00 2021-03-09T10:43:02.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:43:02.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:43:02.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:02.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:02.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:02.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:02.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:02.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:02.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:02.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f8324008870 2021-03-09T10:43:02.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801aeb0 secure :-1 s=READY pgs=10527 cs=0 l=1 rev1=1 rx=0x7f833800e290 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:02.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8324008870 2021-03-09T10:43:02.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.968+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00f300 con 0x7f8324008870 2021-03-09T10:43:02.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:02.972+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f8324008870 2021-03-09T10:43:03.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:43:02 smithi146 bash[13136]: debug 2021-03-09T10:43:02.964+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30668 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:43:07.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.968+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:43:07.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.968+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:07.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.968+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801aeb0 secure :-1 s=READY pgs=10527 cs=0 l=1 rev1=1 rx=0x7f833800e290 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:07.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.968+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801aeb0 secure :-1 s=READY pgs=10527 cs=0 l=1 rev1=1 rx=0x7f833800e290 tx=0x7f833c01b780).stop 2021-03-09T10:43:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.968+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801aeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.968+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801aeb0 unknown :-1 s=CLOSED pgs=10527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.968+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338007b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.968+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.968+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.968+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073140 con 0x7f8324006200 2021-03-09T10:43:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.968+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002d60 con 0x7f8338016f50 2021-03-09T10:43:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.968+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002b60 con 0x7f8324005e00 2021-03-09T10:43:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.968+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:43:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.968+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.968+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:43:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.968+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:07.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.972+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:07.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.972+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:07.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.972+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:07.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.972+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:07.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.972+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:07.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.972+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e2a0 con 0x7f8338016f50 2021-03-09T10:43:07.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.972+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338007b40 secure :-1 s=READY pgs=10538 cs=0 l=1 rev1=1 rx=0x7f83340312a0 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:07.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.976+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833402da60 con 0x7f8338016f50 2021-03-09T10:43:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.976+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402da60 con 0x7f8338016f50 2021-03-09T10:43:07.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:07.980+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833406d180 con 0x7f8338016f50 2021-03-09T10:43:08.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:43:07 smithi146 bash[13136]: debug 2021-03-09T10:43:07.972+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30694 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:43:12.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:43:12.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338007b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:12.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338007b40 secure :-1 s=READY pgs=10538 cs=0 l=1 rev1=1 rx=0x7f83340312a0 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:12.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338007b40 secure :-1 s=READY pgs=10538 cs=0 l=1 rev1=1 rx=0x7f83340312a0 tx=0x7f83340482a0).stop 2021-03-09T10:43:12.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338007b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:12.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338007b40 unknown :-1 s=CLOSED pgs=10538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:12.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:12.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:12.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:12.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801cd10 con 0x7f8324006200 2021-03-09T10:43:12.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024420 con 0x7f8324008870 2021-03-09T10:43:12.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c940 con 0x7f8324005e00 2021-03-09T10:43:12.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:43:12.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:43:12.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:12.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:12.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:12.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:12.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:12.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:12.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:12.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802dd00 con 0x7f8324008870 2021-03-09T10:43:12.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.976+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f160 secure :-1 s=READY pgs=10545 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:12.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.980+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833403bc80 con 0x7f8324008870 2021-03-09T10:43:12.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.980+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833402da60 con 0x7f8324008870 2021-03-09T10:43:12.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:12.984+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833401a570 con 0x7f8324008870 2021-03-09T10:43:13.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:43:12 smithi146 bash[13136]: debug 2021-03-09T10:43:12.976+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30720 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:43:17.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:43:17.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:17.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f160 secure :-1 s=READY pgs=10545 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:17.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f160 secure :-1 s=READY pgs=10545 cs=0 l=1 rev1=1 rx=0x7f833400b800 tx=0x7f83340482a0).stop 2021-03-09T10:43:17.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801f160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:17.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801f160 unknown :-1 s=CLOSED pgs=10545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:17.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:17.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:17.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:17.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c300 con 0x7f8324006200 2021-03-09T10:43:17.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024bb0 con 0x7f8338016f50 2021-03-09T10:43:17.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dce0 con 0x7f8324005e00 2021-03-09T10:43:17.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:43:17.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:17.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:43:17.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:17.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.980+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:17.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.984+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:17.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.984+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:17.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.984+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:17.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.984+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:17.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.984+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065cd0 con 0x7f8338016f50 2021-03-09T10:43:17.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.984+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802dee0 secure :-1 s=READY pgs=10553 cs=0 l=1 rev1=1 rx=0x7f8334056740 tx=0x7f83340482a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:17.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.984+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833403bc80 con 0x7f8338016f50 2021-03-09T10:43:17.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.984+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403bde0 con 0x7f8338016f50 2021-03-09T10:43:17.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:17.988+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334058260 con 0x7f8338016f50 2021-03-09T10:43:18.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:43:17 smithi146 bash[13136]: debug 2021-03-09T10:43:17.980+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30747 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:43:22.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833802dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:43:22.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833802dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:22.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802dee0 secure :-1 s=READY pgs=10553 cs=0 l=1 rev1=1 rx=0x7f8334056740 tx=0x7f83340482a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:22.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802dee0 secure :-1 s=READY pgs=10553 cs=0 l=1 rev1=1 rx=0x7f8334056740 tx=0x7f83340482a0).stop 2021-03-09T10:43:22.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833802dee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:22.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833802dee0 unknown :-1 s=CLOSED pgs=10553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:22.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:22.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:22.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:22.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073140 con 0x7f8324006200 2021-03-09T10:43:22.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:43:22.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806f880 con 0x7f8324008870 2021-03-09T10:43:22.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024420 con 0x7f8324005e00 2021-03-09T10:43:22.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:22.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:43:22.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:22.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:22.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:22.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:22.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:22.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:22.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802dd00 con 0x7f8324008870 2021-03-09T10:43:22.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.988+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380091b0 secure :-1 s=READY pgs=10562 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344041000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:22.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.992+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8324008870 2021-03-09T10:43:22.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.992+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8324008870 2021-03-09T10:43:22.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:22.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834403c280 con 0x7f8324008870 2021-03-09T10:43:23.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:43:22 smithi146 bash[13136]: debug 2021-03-09T10:43:22.988+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30772 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:43:27.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:43:27.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:27.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380091b0 secure :-1 s=READY pgs=10562 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344041000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:27.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380091b0 secure :-1 s=READY pgs=10562 cs=0 l=1 rev1=1 rx=0x7f8344003380 tx=0x7f8344041000).stop 2021-03-09T10:43:27.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380091b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:27.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380091b0 unknown :-1 s=CLOSED pgs=10562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:27.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:27.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:27.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:27.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f8324006200 2021-03-09T10:43:27.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002d60 con 0x7f8338016f50 2021-03-09T10:43:27.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:43:27.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:27.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:43:27.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073140 con 0x7f8324005e00 2021-03-09T10:43:27.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:27.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:27.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:27.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:27.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:27.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:27.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065cd0 con 0x7f8338016f50 2021-03-09T10:43:27.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.992+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f160 secure :-1 s=READY pgs=10570 cs=0 l=1 rev1=1 rx=0x7f833c0077b0 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:27.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa020 con 0x7f8338016f50 2021-03-09T10:43:27.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:27.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338016f50 2021-03-09T10:43:28.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:28.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338016f50 2021-03-09T10:43:28.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:43:27 smithi146 bash[13136]: debug 2021-03-09T10:43:27.992+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30799 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:43:32.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833801f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:43:32.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833801f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:32.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f160 secure :-1 s=READY pgs=10570 cs=0 l=1 rev1=1 rx=0x7f833c0077b0 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:32.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f160 secure :-1 s=READY pgs=10570 cs=0 l=1 rev1=1 rx=0x7f833c0077b0 tx=0x7f833c01b780).stop 2021-03-09T10:43:32.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833801f160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:33.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f160 unknown :-1 s=CLOSED pgs=10570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:33.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:33.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:33.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:33.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017530 con 0x7f8324006200 2021-03-09T10:43:33.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073140 con 0x7f8324008870 2021-03-09T10:43:33.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380131b0 con 0x7f8324005e00 2021-03-09T10:43:33.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:43:33.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:33.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:33.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:43:33.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:33.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:33.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:33.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:33.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:33.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:32.996+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802dd00 con 0x7f8324008870 2021-03-09T10:43:33.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:33.000+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380091b0 secure :-1 s=READY pgs=10580 cs=0 l=1 rev1=1 rx=0x7f834403d040 tx=0x7f8344041000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:33.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:33.000+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f8324008870 2021-03-09T10:43:33.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:33.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f8324008870 2021-03-09T10:43:33.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:33.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405bd70 con 0x7f8324008870 2021-03-09T10:43:33.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:43:33 smithi146 bash[13136]: debug 2021-03-09T10:43:32.996+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30824 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:43:38.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:43:38.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:38.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380091b0 secure :-1 s=READY pgs=10580 cs=0 l=1 rev1=1 rx=0x7f834403d040 tx=0x7f8344041000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:38.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380091b0 secure :-1 s=READY pgs=10580 cs=0 l=1 rev1=1 rx=0x7f834403d040 tx=0x7f8344041000).stop 2021-03-09T10:43:38.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380091b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:38.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380091b0 unknown :-1 s=CLOSED pgs=10580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:38.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:38.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:38.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:38.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dce0 con 0x7f8324006200 2021-03-09T10:43:38.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f8338016f50 2021-03-09T10:43:38.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073140 con 0x7f8324005e00 2021-03-09T10:43:38.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:43:38.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:43:38.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:38.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:38.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:38.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:38.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:38.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:38.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:38.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026240 con 0x7f8338016f50 2021-03-09T10:43:38.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.004+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f800 secure :-1 s=READY pgs=10589 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344041000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:38.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344066690 con 0x7f8338016f50 2021-03-09T10:43:38.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.008+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033070 con 0x7f8338016f50 2021-03-09T10:43:38.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:38.012+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405d1d0 con 0x7f8338016f50 2021-03-09T10:43:38.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:43:38 smithi146 bash[13136]: debug 2021-03-09T10:43:38.004+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30852 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:43:43.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833801f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:43:43.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833801f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:43.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f800 secure :-1 s=READY pgs=10589 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344041000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:43.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f800 secure :-1 s=READY pgs=10589 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344041000).stop 2021-03-09T10:43:43.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833801f800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:43.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833801f800 unknown :-1 s=CLOSED pgs=10589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:43.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:43.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:43.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:43.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801cd10 con 0x7f8324006200 2021-03-09T10:43:43.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324008870 2021-03-09T10:43:43.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806f030 con 0x7f8324005e00 2021-03-09T10:43:43.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:43:43.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:43.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:43:43.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:43.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:43.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:43.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:43.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:43.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:43.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338002b10 con 0x7f8324008870 2021-03-09T10:43:43.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.011+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800c560 secure :-1 s=READY pgs=10596 cs=0 l=1 rev1=1 rx=0x7f8334004230 tx=0x7f833403ba30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:43.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.015+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400b980 con 0x7f8324008870 2021-03-09T10:43:43.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.019+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334000bc0 con 0x7f8324008870 2021-03-09T10:43:43.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:43.019+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833403bd50 con 0x7f8324008870 2021-03-09T10:43:43.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:43:43 smithi146 bash[13136]: debug 2021-03-09T10:43:43.012+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30877 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:43:48.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.015+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:43:48.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.015+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:48.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.015+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800c560 secure :-1 s=READY pgs=10596 cs=0 l=1 rev1=1 rx=0x7f8334004230 tx=0x7f833403ba30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:48.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.015+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800c560 secure :-1 s=READY pgs=10596 cs=0 l=1 rev1=1 rx=0x7f8334004230 tx=0x7f833403ba30).stop 2021-03-09T10:43:48.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.015+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800c560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:48.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.015+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800c560 unknown :-1 s=CLOSED pgs=10596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:48.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.015+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:48.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.015+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338061aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:48.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.015+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:48.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.015+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074e10 con 0x7f8324006200 2021-03-09T10:43:48.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.015+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f8338016f50 2021-03-09T10:43:48.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.015+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:43:48.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.015+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380131b0 con 0x7f8324005e00 2021-03-09T10:43:48.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.015+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:48.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.015+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:43:48.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.015+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:48.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.019+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338061aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:48.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.019+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:48.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.019+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:48.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.019+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:48.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.019+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:48.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.019+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065440 con 0x7f8338016f50 2021-03-09T10:43:48.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.019+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338061aa0 secure :-1 s=READY pgs=10605 cs=0 l=1 rev1=1 rx=0x7f8344017040 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:48.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.023+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344057330 con 0x7f8338016f50 2021-03-09T10:43:48.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.023+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404e500 con 0x7f8338016f50 2021-03-09T10:43:48.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:48.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338016f50 2021-03-09T10:43:48.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:43:48 smithi146 bash[13136]: debug 2021-03-09T10:43:48.016+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30904 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:43:53.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338061aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:43:53.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338061aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:53.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338061aa0 secure :-1 s=READY pgs=10605 cs=0 l=1 rev1=1 rx=0x7f8344017040 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:53.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338061aa0 secure :-1 s=READY pgs=10605 cs=0 l=1 rev1=1 rx=0x7f8344017040 tx=0x7f8344003f90).stop 2021-03-09T10:43:53.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f8338061aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:53.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f8338061aa0 unknown :-1 s=CLOSED pgs=10605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:53.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:53.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:53.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:53.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f8324006200 2021-03-09T10:43:53.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f8324008870 2021-03-09T10:43:53.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801cd10 con 0x7f8324005e00 2021-03-09T10:43:53.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:43:53.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:53.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:43:53.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:53.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:53.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:53.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:53.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:53.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:53.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ac50 con 0x7f8324008870 2021-03-09T10:43:53.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.023+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800c560 secure :-1 s=READY pgs=10614 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344030000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:53.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405fb00 con 0x7f8324008870 2021-03-09T10:43:53.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344033050 con 0x7f8324008870 2021-03-09T10:43:53.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:53.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8324008870 2021-03-09T10:43:53.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:43:53 smithi146 bash[13136]: debug 2021-03-09T10:43:53.024+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30929 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:43:58.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.027+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:43:58.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.027+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:58.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.027+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800c560 secure :-1 s=READY pgs=10614 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344030000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:58.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.027+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800c560 secure :-1 s=READY pgs=10614 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344030000).stop 2021-03-09T10:43:58.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800c560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:58.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.027+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800c560 unknown :-1 s=CLOSED pgs=10614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:58.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.027+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:58.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.027+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833800a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:58.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.027+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:58.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.027+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:43:58.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.027+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:58.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.027+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324006200 2021-03-09T10:43:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.027+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:43:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024420 con 0x7f8338016f50 2021-03-09T10:43:58.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dce0 con 0x7f8324005e00 2021-03-09T10:43:58.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 msgr2=0x7f833801d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-03-09T10:43:58.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:58.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833800a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:58.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324005e00 0x7f833801d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:58.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 msgr2=0x7f833800a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T10:43:58.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338016f50 0x7f833800a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:58.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:58.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006200 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:58.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f832400ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:58.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f83240053d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:58.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83240072e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:58.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324006e60 2021-03-09T10:43:58.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024420 con 0x7f83240069e0 2021-03-09T10:43:58.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dce0 con 0x7f8324006600 2021-03-09T10:43:58.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83240072e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:43:58.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83240072e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:58.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f83240053d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:58.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f832400ad00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:43:58.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f832400ad00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:58.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f832400ad00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:58.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.031+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f832400ad00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:58.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.035+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83240072e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:58.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.035+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83240072e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:58.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.035+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017d20 con 0x7f83240069e0 2021-03-09T10:43:58.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.035+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f83240053d0 secure :-1 s=READY pgs=10625 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344030000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:58.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440662c0 con 0x7f83240069e0 2021-03-09T10:43:58.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f83240069e0 2021-03-09T10:43:58.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:43:58.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344010250 con 0x7f83240069e0 2021-03-09T10:43:58.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:43:58 smithi146 bash[13136]: debug 2021-03-09T10:43:58.032+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30957 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:44:03.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f83240053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:44:03.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f83240053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:03.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f83240053d0 secure :-1 s=READY pgs=10625 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344030000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:03.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f83240053d0 secure :-1 s=READY pgs=10625 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344030000).stop 2021-03-09T10:44:03.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f83240053d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:03.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f83240053d0 unknown :-1 s=CLOSED pgs=10625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:03.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:03.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:03.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:03.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024fa0 con 0x7f8324006e60 2021-03-09T10:44:03.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024420 con 0x7f8324008870 2021-03-09T10:44:03.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020570 con 0x7f8324006600 2021-03-09T10:44:03.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:44:03.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:03.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:44:03.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:03.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:03.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:03.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:03.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:03.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:03.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.035+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026240 con 0x7f8324008870 2021-03-09T10:44:03.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016a60 secure :-1 s=READY pgs=10631 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:03.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa040 con 0x7f8324008870 2021-03-09T10:44:03.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8324008870 2021-03-09T10:44:03.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:03.043+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0613a0 con 0x7f8324008870 2021-03-09T10:44:03.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:44:03 smithi146 bash[13136]: debug 2021-03-09T10:44:03.036+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 30982 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:44:08.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338016a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:44:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338016a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016a60 secure :-1 s=READY pgs=10631 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016a60 secure :-1 s=READY pgs=10631 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01b780).stop 2021-03-09T10:44:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338016a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:08.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338016a60 unknown :-1 s=CLOSED pgs=10631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:08.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:08.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338026520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:08.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:08.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dce0 con 0x7f8324006e60 2021-03-09T10:44:08.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024420 con 0x7f83240069e0 2021-03-09T10:44:08.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065d20 con 0x7f8324006600 2021-03-09T10:44:08.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:44:08.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:08.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:44:08.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:08.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.039+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338026520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:08.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.043+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:08.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.043+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:08.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.043+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:08.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.043+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:08.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.043+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802cb40 con 0x7f83240069e0 2021-03-09T10:44:08.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.043+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338026520 secure :-1 s=READY pgs=10640 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344030000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:08.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f83240069e0 2021-03-09T10:44:08.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f83240069e0 2021-03-09T10:44:08.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:08.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344010b00 con 0x7f83240069e0 2021-03-09T10:44:08.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:44:08 smithi146 bash[13136]: debug 2021-03-09T10:44:08.043+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31010 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:44:13.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338026520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:44:13.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338026520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:13.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338026520 secure :-1 s=READY pgs=10640 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344030000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:13.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338026520 secure :-1 s=READY pgs=10640 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344030000).stop 2021-03-09T10:44:13.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338026520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:13.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338026520 unknown :-1 s=CLOSED pgs=10640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:13.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:13.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:13.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:13.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020570 con 0x7f8324006e60 2021-03-09T10:44:13.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074750 con 0x7f8324008870 2021-03-09T10:44:13.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024420 con 0x7f8324006600 2021-03-09T10:44:13.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:44:13.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:13.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:44:13.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:13.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:13.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:13.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:13.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:13.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:13.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380294c0 con 0x7f8324008870 2021-03-09T10:44:13.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.047+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017150 secure :-1 s=READY pgs=10649 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833403ba30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:13.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833403baa0 con 0x7f8324008870 2021-03-09T10:44:13.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833403e490 con 0x7f8324008870 2021-03-09T10:44:13.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:13.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833402d300 con 0x7f8324008870 2021-03-09T10:44:13.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:44:13 smithi146 bash[13136]: debug 2021-03-09T10:44:13.047+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31035 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:44:18.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338017150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:44:18.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338017150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:18.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017150 secure :-1 s=READY pgs=10649 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833403ba30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:18.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017150 secure :-1 s=READY pgs=10649 cs=0 l=1 rev1=1 rx=0x7f833400bfd0 tx=0x7f833403ba30).stop 2021-03-09T10:44:18.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f8338017150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:18.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f8338017150 unknown :-1 s=CLOSED pgs=10649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:18.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:18.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:18.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338060780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:18.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801acf0 con 0x7f8324006e60 2021-03-09T10:44:18.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380131b0 con 0x7f83240069e0 2021-03-09T10:44:18.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011020 con 0x7f8324006600 2021-03-09T10:44:18.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:44:18.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:18.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:44:18.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:18.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.051+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338060780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:18.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:18.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:18.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:18.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:18.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338002b10 con 0x7f83240069e0 2021-03-09T10:44:18.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338060780 secure :-1 s=READY pgs=10657 cs=0 l=1 rev1=1 rx=0x7f833c079a70 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:18.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f83240069e0 2021-03-09T10:44:18.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f83240069e0 2021-03-09T10:44:18.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:18.059+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c290 con 0x7f83240069e0 2021-03-09T10:44:18.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:44:18 smithi146 bash[13136]: debug 2021-03-09T10:44:18.055+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31063 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:44:23.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338060780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2021-03-09T10:44:23.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.055+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338060780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:23.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338060780 secure :-1 s=READY pgs=10657 cs=0 l=1 rev1=1 rx=0x7f833c079a70 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:23.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.055+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338060780 secure :-1 s=READY pgs=10657 cs=0 l=1 rev1=1 rx=0x7f833c079a70 tx=0x7f833c01b780).stop 2021-03-09T10:44:23.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338060780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:23.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.055+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338060780 unknown :-1 s=CLOSED pgs=10657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:23.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.055+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:23.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.055+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:23.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.055+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:23.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324006e60 2021-03-09T10:44:23.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024420 con 0x7f8324008870 2021-03-09T10:44:23.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.055+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801acf0 con 0x7f8324006600 2021-03-09T10:44:23.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.055+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:44:23.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.055+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:23.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.055+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:44:23.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.055+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:23.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.059+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:23.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.059+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:23.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.059+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:23.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.059+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:23.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.059+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:23.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.059+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028fe0 con 0x7f8324008870 2021-03-09T10:44:23.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.059+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a2f0 secure :-1 s=READY pgs=10669 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:23.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8324008870 2021-03-09T10:44:23.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0a1710 con 0x7f8324008870 2021-03-09T10:44:23.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:23.067+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01b290 con 0x7f8324008870 2021-03-09T10:44:23.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:44:23 smithi146 bash[13136]: debug 2021-03-09T10:44:23.059+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31088 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:44:28.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:44:28.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:28.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a2f0 secure :-1 s=READY pgs=10669 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:28.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a2f0 secure :-1 s=READY pgs=10669 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c01b780).stop 2021-03-09T10:44:28.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833800a2f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:28.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833800a2f0 unknown :-1 s=CLOSED pgs=10669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:28.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:28.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338061140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:28.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:28.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380745f0 con 0x7f8324006e60 2021-03-09T10:44:28.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380131b0 con 0x7f83240069e0 2021-03-09T10:44:28.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d4e0 con 0x7f8324006600 2021-03-09T10:44:28.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:44:28.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:44:28.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:28.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:28.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338061140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:28.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:28.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:28.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:28.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:28.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.063+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802cb40 con 0x7f83240069e0 2021-03-09T10:44:28.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.067+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338061140 secure :-1 s=READY pgs=10675 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344030000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:28.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.067+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405d9c0 con 0x7f83240069e0 2021-03-09T10:44:28.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.071+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f83240069e0 2021-03-09T10:44:28.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:28.075+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344075340 con 0x7f83240069e0 2021-03-09T10:44:28.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:44:28 smithi146 bash[13136]: debug 2021-03-09T10:44:28.063+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31116 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:44:33.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338061140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:44:33.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338061140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:33.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338061140 secure :-1 s=READY pgs=10675 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344030000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:33.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338061140 secure :-1 s=READY pgs=10675 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344030000).stop 2021-03-09T10:44:33.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338061140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:33.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338061140 unknown :-1 s=CLOSED pgs=10675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:33.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:33.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:33.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:33.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380233e0 con 0x7f8324006e60 2021-03-09T10:44:33.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800f760 con 0x7f8324008870 2021-03-09T10:44:33.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324006600 2021-03-09T10:44:33.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:44:33.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:33.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:44:33.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:33.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.066+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:33.071 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.070+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:33.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.070+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:33.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.070+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:33.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.070+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:33.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.070+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ddb0 con 0x7f8324008870 2021-03-09T10:44:33.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.070+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801b0f0 secure :-1 s=READY pgs=10683 cs=0 l=1 rev1=1 rx=0x7f8338068720 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:33.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.074+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c072ef0 con 0x7f8324008870 2021-03-09T10:44:33.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.074+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0266f0 con 0x7f8324008870 2021-03-09T10:44:33.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:33.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8324008870 2021-03-09T10:44:33.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:44:33 smithi146 bash[13136]: debug 2021-03-09T10:44:33.071+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31141 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:44:38.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.070+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:44:38.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.070+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:38.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.070+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801b0f0 secure :-1 s=READY pgs=10683 cs=0 l=1 rev1=1 rx=0x7f8338068720 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:38.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.070+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801b0f0 secure :-1 s=READY pgs=10683 cs=0 l=1 rev1=1 rx=0x7f8338068720 tx=0x7f833c01b780).stop 2021-03-09T10:44:38.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.070+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f833801b0f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:38.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.070+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f833801b0f0 unknown :-1 s=CLOSED pgs=10683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:38.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.070+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:38.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.070+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:38.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.070+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338061aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:38.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.070+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064ee0 con 0x7f8324006e60 2021-03-09T10:44:38.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.070+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380233e0 con 0x7f83240069e0 2021-03-09T10:44:38.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.070+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338075260 con 0x7f8324006600 2021-03-09T10:44:38.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.070+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:44:38.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.074+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:38.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.074+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:44:38.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.074+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338061aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:38.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.074+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:38.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.074+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:38.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.074+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:38.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.074+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:38.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.074+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:38.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.074+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f83240069e0 2021-03-09T10:44:38.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.074+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338061aa0 secure :-1 s=READY pgs=10692 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:38.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f83240069e0 2021-03-09T10:44:38.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c02f5c0 con 0x7f83240069e0 2021-03-09T10:44:38.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:38.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f83240069e0 2021-03-09T10:44:38.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:44:38 smithi146 bash[13136]: debug 2021-03-09T10:44:38.075+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31169 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:44:43.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338061aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:44:43.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338061aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:43.079 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338061aa0 secure :-1 s=READY pgs=10692 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:43.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338061aa0 secure :-1 s=READY pgs=10692 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c01b780).stop 2021-03-09T10:44:43.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338061aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:43.080 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338061aa0 unknown :-1 s=CLOSED pgs=10692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:43.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380218f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:43.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:43.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:43.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e7c0 con 0x7f8324006e60 2021-03-09T10:44:43.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002b60 con 0x7f8324008870 2021-03-09T10:44:43.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065d20 con 0x7f8324006600 2021-03-09T10:44:43.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380218f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:43.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:44:43.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:44:43.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:43.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:43.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:43.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:43.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:43.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:43.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.078+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802dd00 con 0x7f8324008870 2021-03-09T10:44:43.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.082+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380218f0 secure :-1 s=READY pgs=10700 cs=0 l=1 rev1=1 rx=0x7f8334064ac0 tx=0x7f833403e600).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:43.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334064220 con 0x7f8324008870 2021-03-09T10:44:43.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8334064220 con 0x7f8324008870 2021-03-09T10:44:43.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:43.086+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8334064dc0 con 0x7f8324008870 2021-03-09T10:44:43.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:44:43 smithi146 bash[13136]: debug 2021-03-09T10:44:43.079+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31194 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:44:48.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380218f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:44:48.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380218f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:48.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380218f0 secure :-1 s=READY pgs=10700 cs=0 l=1 rev1=1 rx=0x7f8334064ac0 tx=0x7f833403e600).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:48.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380218f0 secure :-1 s=READY pgs=10700 cs=0 l=1 rev1=1 rx=0x7f8334064ac0 tx=0x7f833403e600).stop 2021-03-09T10:44:48.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 msgr2=0x7f83380218f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:48.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324008870 0x7f83380218f0 unknown :-1 s=CLOSED pgs=10700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:48.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:48.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:48.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338067ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:48.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380131b0 con 0x7f8324006e60 2021-03-09T10:44:48.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f83240069e0 2021-03-09T10:44:48.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ab70 con 0x7f8324006600 2021-03-09T10:44:48.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f83427fc700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:44:48.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:48.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:44:48.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:48.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.082+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338067ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:48.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.086+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:48.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.086+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:48.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.086+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:48.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.086+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:48.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.086+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028fe0 con 0x7f83240069e0 2021-03-09T10:44:48.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.086+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338067ee0 secure :-1 s=READY pgs=10709 cs=0 l=1 rev1=1 rx=0x7f833c0108d0 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:48.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c007c00 con 0x7f83240069e0 2021-03-09T10:44:48.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c010940 con 0x7f83240069e0 2021-03-09T10:44:48.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:48.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f83240069e0 2021-03-09T10:44:48.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:44:48 smithi146 bash[13136]: debug 2021-03-09T10:44:48.087+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31222 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:44:50.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:50.646+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2021-03-09T10:44:50.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:50.646+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:50.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:50.646+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c06e000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:50.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:50.646+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f833c012490 tx=0x7f833c06e000).stop 2021-03-09T10:44:50.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:50.646+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T10:44:50.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:50.646+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:50.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:50.646+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:50.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:50.646+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8324008870 0x7f83380683d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:50.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:50.646+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> v2:172.21.15.5:6800/4269212764 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8334063180 con 0x7f8324008870 2021-03-09T10:44:50.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:50.646+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8324008870 0x7f83380683d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:50.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:50.646+0000 7f83427fc700 1 --2- 172.21.15.5:0/3543657960 >> v2:172.21.15.5:6800/4269212764 conn(0x7f8324008870 0x7f83380683d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f8334004ca0 tx=0x7f833403e600).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:53.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:44:53.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:53.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338067ee0 secure :-1 s=READY pgs=10709 cs=0 l=1 rev1=1 rx=0x7f833c0108d0 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:53.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338067ee0 secure :-1 s=READY pgs=10709 cs=0 l=1 rev1=1 rx=0x7f833c0108d0 tx=0x7f833c01b780).stop 2021-03-09T10:44:53.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338067ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:53.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338067ee0 unknown :-1 s=CLOSED pgs=10709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:53.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:53.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:53.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:53.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d690 con 0x7f8324006e60 2021-03-09T10:44:53.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8338071d10 2021-03-09T10:44:53.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324006600 2021-03-09T10:44:53.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:44:53.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:53.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:53.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:44:53.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:53.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:53.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:53.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:53.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:53.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802dd00 con 0x7f8338071d10 2021-03-09T10:44:53.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.090+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10718 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f8344004f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:53.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f8338071d10 2021-03-09T10:44:53.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344042ec0 con 0x7f8338071d10 2021-03-09T10:44:53.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:53.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338071d10 2021-03-09T10:44:53.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:44:53 smithi146 bash[13136]: debug 2021-03-09T10:44:53.091+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31247 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:44:58.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:44:58.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:58.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10718 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f8344004f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:58.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10718 cs=0 l=1 rev1=1 rx=0x7f8344004310 tx=0x7f8344004f90).stop 2021-03-09T10:44:58.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:58.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=10718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:58.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:58.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338002280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:58.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:58.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338077130 con 0x7f8324006e60 2021-03-09T10:44:58.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a90 con 0x7f83240069e0 2021-03-09T10:44:58.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801cd10 con 0x7f8324006600 2021-03-09T10:44:58.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338002280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:58.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:44:58.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:58.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:44:58.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.094+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:58.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.098+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:58.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.098+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:58.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.098+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:58.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.098+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:58.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.098+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801a9e0 con 0x7f83240069e0 2021-03-09T10:44:58.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.098+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338002280 secure :-1 s=READY pgs=10726 cs=0 l=1 rev1=1 rx=0x7f834404b490 tx=0x7f8344004f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:58.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404b500 con 0x7f83240069e0 2021-03-09T10:44:58.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.102+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404b500 con 0x7f83240069e0 2021-03-09T10:44:58.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:44:58.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405edc0 con 0x7f83240069e0 2021-03-09T10:44:58.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:44:58 smithi146 bash[13136]: debug 2021-03-09T10:44:58.099+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31274 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:45:03.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.098+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338002280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:45:03.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.098+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338002280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:03.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.098+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338002280 secure :-1 s=READY pgs=10726 cs=0 l=1 rev1=1 rx=0x7f834404b490 tx=0x7f8344004f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:03.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.098+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338002280 secure :-1 s=READY pgs=10726 cs=0 l=1 rev1=1 rx=0x7f834404b490 tx=0x7f8344004f90).stop 2021-03-09T10:45:03.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338002280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:03.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.098+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338002280 unknown :-1 s=CLOSED pgs=10726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:03.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.098+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:03.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.098+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:03.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.098+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:03.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ab70 con 0x7f8324006e60 2021-03-09T10:45:03.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008610 con 0x7f8338071d10 2021-03-09T10:45:03.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.098+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018820 con 0x7f8324006600 2021-03-09T10:45:03.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.102+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:45:03.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.102+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:03.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.102+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:45:03.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.102+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:03.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.102+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:03.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.102+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:03.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.102+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:03.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.102+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:03.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.102+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:03.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.102+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380643d0 con 0x7f8338071d10 2021-03-09T10:45:03.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.102+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10736 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:03.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa080 con 0x7f8338071d10 2021-03-09T10:45:03.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079630 con 0x7f8338071d10 2021-03-09T10:45:03.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:03.110+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007070 con 0x7f8338071d10 2021-03-09T10:45:03.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:45:03 smithi146 bash[13136]: debug 2021-03-09T10:45:03.103+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31300 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:45:08.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:45:08.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:08.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10736 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:08.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10736 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c01b780).stop 2021-03-09T10:45:08.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:08.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=10736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338017150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380745f0 con 0x7f8324006e60 2021-03-09T10:45:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069fb0 con 0x7f83240069e0 2021-03-09T10:45:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324006600 2021-03-09T10:45:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:45:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:45:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.106+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338017150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.110+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.110+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.110+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.110+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.110+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380287e0 con 0x7f83240069e0 2021-03-09T10:45:08.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.110+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338017150 secure :-1 s=READY pgs=10745 cs=0 l=1 rev1=1 rx=0x7f83440579e0 tx=0x7f8344004f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:08.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.114+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344042ec0 con 0x7f83240069e0 2021-03-09T10:45:08.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.114+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f83240069e0 2021-03-09T10:45:08.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:08.118+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344075340 con 0x7f83240069e0 2021-03-09T10:45:08.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:45:08 smithi146 bash[13136]: debug 2021-03-09T10:45:08.111+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31327 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:45:13.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.110+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338017150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:45:13.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.110+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338017150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:13.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.110+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338017150 secure :-1 s=READY pgs=10745 cs=0 l=1 rev1=1 rx=0x7f83440579e0 tx=0x7f8344004f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:13.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.110+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338017150 secure :-1 s=READY pgs=10745 cs=0 l=1 rev1=1 rx=0x7f83440579e0 tx=0x7f8344004f90).stop 2021-03-09T10:45:13.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.110+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338017150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:13.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.110+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338017150 unknown :-1 s=CLOSED pgs=10745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:13.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:13.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:13.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:13.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a200 con 0x7f8324006e60 2021-03-09T10:45:13.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338077130 con 0x7f8338071d10 2021-03-09T10:45:13.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e1c0 con 0x7f8324006600 2021-03-09T10:45:13.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:45:13.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:13.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:13.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:45:13.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:13.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:13.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:13.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:13.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:13.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380668c0 con 0x7f8338071d10 2021-03-09T10:45:13.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.114+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10752 cs=0 l=1 rev1=1 rx=0x7f833c0546b0 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:13.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.118+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338071d10 2021-03-09T10:45:13.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.118+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f8338071d10 2021-03-09T10:45:13.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:13.122+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0792c0 con 0x7f8338071d10 2021-03-09T10:45:13.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:45:13 smithi146 bash[13136]: debug 2021-03-09T10:45:13.114+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31353 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:45:18.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:45:18.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:18.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10752 cs=0 l=1 rev1=1 rx=0x7f833c0546b0 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:18.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10752 cs=0 l=1 rev1=1 rx=0x7f833c0546b0 tx=0x7f833c01b780).stop 2021-03-09T10:45:18.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:18.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=10752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:18.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:18.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:18.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338017150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:18.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380626c0 con 0x7f8324006e60 2021-03-09T10:45:18.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c300 con 0x7f83240069e0 2021-03-09T10:45:18.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324006600 2021-03-09T10:45:18.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:45:18.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:18.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:45:18.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:18.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.117+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338017150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:18.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.121+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:18.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.121+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:18.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.121+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:18.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.121+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:18.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.121+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380637d0 con 0x7f83240069e0 2021-03-09T10:45:18.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.121+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338017150 secure :-1 s=READY pgs=10763 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:18.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.125+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c079800 con 0x7f83240069e0 2021-03-09T10:45:18.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.125+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0546e0 con 0x7f83240069e0 2021-03-09T10:45:18.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:18.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004070 con 0x7f83240069e0 2021-03-09T10:45:18.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:45:18 smithi146 bash[13136]: debug 2021-03-09T10:45:18.118+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31380 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:45:23.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.121+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338017150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:45:23.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.121+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338017150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:23.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.121+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338017150 secure :-1 s=READY pgs=10763 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:23.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.121+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338017150 secure :-1 s=READY pgs=10763 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c01b780).stop 2021-03-09T10:45:23.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.121+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f8338017150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:23.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.121+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f8338017150 unknown :-1 s=CLOSED pgs=10763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:23.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.125+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:23.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.125+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:23.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.125+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:23.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.125+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800acf0 con 0x7f8324006e60 2021-03-09T10:45:23.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.125+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801efb0 con 0x7f8338071d10 2021-03-09T10:45:23.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.125+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e1c0 con 0x7f8324006600 2021-03-09T10:45:23.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.125+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:45:23.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.125+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:23.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.125+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:45:23.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.125+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:23.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.125+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:23.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.125+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:23.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.125+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:23.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.129+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:23.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.129+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:23.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.129+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380668c0 con 0x7f8338071d10 2021-03-09T10:45:23.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.129+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10775 cs=0 l=1 rev1=1 rx=0x7f833c0299c0 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:23.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0754c0 con 0x7f8338071d10 2021-03-09T10:45:23.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0a0ee0 con 0x7f8338071d10 2021-03-09T10:45:23.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:23.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01ab20 con 0x7f8338071d10 2021-03-09T10:45:23.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:45:23 smithi146 bash[13136]: debug 2021-03-09T10:45:23.126+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31407 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:45:28.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:45:28.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:28.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10775 cs=0 l=1 rev1=1 rx=0x7f833c0299c0 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:28.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10775 cs=0 l=1 rev1=1 rx=0x7f833c0299c0 tx=0x7f833c01b780).stop 2021-03-09T10:45:28.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:28.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=10775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:28.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:28.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833801f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:28.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:28.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f8324006e60 2021-03-09T10:45:28.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801eb60 con 0x7f83240069e0 2021-03-09T10:45:28.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f8324006600 2021-03-09T10:45:28.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:45:28.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:28.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:45:28.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:28.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833801f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:28.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:28.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:28.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:28.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:28.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e7c0 con 0x7f83240069e0 2021-03-09T10:45:28.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.129+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833801f7c0 secure :-1 s=READY pgs=10778 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344004f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:28.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f83240069e0 2021-03-09T10:45:28.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.133+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344020170 con 0x7f83240069e0 2021-03-09T10:45:28.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:28.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402adb0 con 0x7f83240069e0 2021-03-09T10:45:28.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:45:28 smithi146 bash[13136]: debug 2021-03-09T10:45:28.130+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31433 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:45:33.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833801f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:45:33.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833801f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:33.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833801f7c0 secure :-1 s=READY pgs=10778 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344004f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:33.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833801f7c0 secure :-1 s=READY pgs=10778 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344004f90).stop 2021-03-09T10:45:33.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833801f7c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:33.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833801f7c0 unknown :-1 s=CLOSED pgs=10778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:33.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:33.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:33.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:33.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324006e60 2021-03-09T10:45:33.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800b4c0 con 0x7f8338071d10 2021-03-09T10:45:33.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338015aa0 con 0x7f8324006600 2021-03-09T10:45:33.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:45:33.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:45:33.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:33.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:33.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:33.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:33.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:33.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:33.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:33.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e5c0 con 0x7f8338071d10 2021-03-09T10:45:33.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.137+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10787 cs=0 l=1 rev1=1 rx=0x7f833c041af0 tx=0x7f833c01b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:33.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0a0ee0 con 0x7f8338071d10 2021-03-09T10:45:33.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f8338071d10 2021-03-09T10:45:33.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:33.145+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0792c0 con 0x7f8338071d10 2021-03-09T10:45:33.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:45:33 smithi146 bash[13136]: debug 2021-03-09T10:45:33.138+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31460 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:45:38.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:45:38.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:38.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10787 cs=0 l=1 rev1=1 rx=0x7f833c041af0 tx=0x7f833c01b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:38.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10787 cs=0 l=1 rev1=1 rx=0x7f833c041af0 tx=0x7f833c01b780).stop 2021-03-09T10:45:38.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:38.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=10787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:38.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:38.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:38.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:38.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010830 con 0x7f8324006e60 2021-03-09T10:45:38.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e1c0 con 0x7f83240069e0 2021-03-09T10:45:38.146 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324006600 2021-03-09T10:45:38.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:45:38.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:38.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:45:38.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:38.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.141+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:38.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.145+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:38.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.145+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:38.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.145+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:38.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.145+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:38.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.145+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338069f60 con 0x7f83240069e0 2021-03-09T10:45:38.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.145+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802cd20 secure :-1 s=READY pgs=10798 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:38.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.145+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344070eb0 con 0x7f83240069e0 2021-03-09T10:45:38.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.153+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834403cef0 con 0x7f83240069e0 2021-03-09T10:45:38.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:38.153+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f83240069e0 2021-03-09T10:45:38.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:45:38 smithi146 bash[13136]: debug 2021-03-09T10:45:38.142+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31486 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:45:43.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.149+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:45:43.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.149+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:43.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.149+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802cd20 secure :-1 s=READY pgs=10798 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:43.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.149+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802cd20 secure :-1 s=READY pgs=10798 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004f90).stop 2021-03-09T10:45:43.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802cd20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:43.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.149+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802cd20 unknown :-1 s=CLOSED pgs=10798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:43.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.149+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:43.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.149+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:43.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.149+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:43.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801cd10 con 0x7f8324006e60 2021-03-09T10:45:43.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f8338071d10 2021-03-09T10:45:43.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.149+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066090 con 0x7f8324006600 2021-03-09T10:45:43.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.149+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:45:43.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.149+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:45:43.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.149+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:43.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.149+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:43.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.157+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:43.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.157+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:43.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.157+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:43.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.157+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:43.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.157+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:43.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.157+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380185f0 con 0x7f8338071d10 2021-03-09T10:45:43.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.157+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10810 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c058f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:43.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338071d10 2021-03-09T10:45:43.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079630 con 0x7f8338071d10 2021-03-09T10:45:43.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:43.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338071d10 2021-03-09T10:45:43.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:45:43 smithi146 bash[13136]: debug 2021-03-09T10:45:43.150+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31513 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:45:48.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:45:48.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:48.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10810 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c058f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:48.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10810 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c058f90).stop 2021-03-09T10:45:48.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:48.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=10810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:48.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:48.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:48.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:48.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a200 con 0x7f8324006e60 2021-03-09T10:45:48.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801efb0 con 0x7f83240069e0 2021-03-09T10:45:48.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e1c0 con 0x7f8324006600 2021-03-09T10:45:48.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:45:48.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:48.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:45:48.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:48.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.153+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:48.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.157+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:48.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.157+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:48.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.157+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:48.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.157+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:48.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.157+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801fb50 con 0x7f83240069e0 2021-03-09T10:45:48.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.157+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10816 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c058f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:48.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f83240069e0 2021-03-09T10:45:48.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.157+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05c380 con 0x7f83240069e0 2021-03-09T10:45:48.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:48.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c072390 con 0x7f83240069e0 2021-03-09T10:45:48.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:45:48 smithi146 bash[13136]: debug 2021-03-09T10:45:48.154+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31539 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:45:53.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:45:53.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:53.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10816 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c058f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:53.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10816 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c058f90).stop 2021-03-09T10:45:53.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:53.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=10816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:53.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:53.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:53.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:53.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074aa0 con 0x7f8324006e60 2021-03-09T10:45:53.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338067360 con 0x7f8338071d10 2021-03-09T10:45:53.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062720 con 0x7f8324006600 2021-03-09T10:45:53.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:45:53.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:53.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:45:53.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:53.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:53.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:53.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:53.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:53.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:53.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380735e0 con 0x7f8338071d10 2021-03-09T10:45:53.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.161+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10822 cs=0 l=1 rev1=1 rx=0x7f83380692d0 tx=0x7f8344004f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:53.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403cef0 con 0x7f8338071d10 2021-03-09T10:45:53.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834403cef0 con 0x7f8338071d10 2021-03-09T10:45:53.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:53.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338071d10 2021-03-09T10:45:53.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:45:53 smithi146 bash[13136]: debug 2021-03-09T10:45:53.162+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31566 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:45:58.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:45:58.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:58.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10822 cs=0 l=1 rev1=1 rx=0x7f83380692d0 tx=0x7f8344004f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:58.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10822 cs=0 l=1 rev1=1 rx=0x7f83380692d0 tx=0x7f8344004f90).stop 2021-03-09T10:45:58.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:58.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=10822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:58.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:58.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:58.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:58.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008be0 con 0x7f8324006e60 2021-03-09T10:45:58.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f83240069e0 2021-03-09T10:45:58.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e1c0 con 0x7f8324006600 2021-03-09T10:45:58.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:45:58.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:58.168 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:58.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:45:58.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:58.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028fa0 con 0x7f83240069e0 2021-03-09T10:45:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.165+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10831 cs=0 l=1 rev1=1 rx=0x7f833c0258b0 tx=0x7f833c058f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:58.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f83240069e0 2021-03-09T10:45:58.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.169+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0252e0 con 0x7f83240069e0 2021-03-09T10:45:58.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:45:58.173+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f83240069e0 2021-03-09T10:45:58.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:45:58 smithi146 bash[13136]: debug 2021-03-09T10:45:58.166+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31592 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:46:03.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:46:03.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:03.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10831 cs=0 l=1 rev1=1 rx=0x7f833c0258b0 tx=0x7f833c058f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:03.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10831 cs=0 l=1 rev1=1 rx=0x7f833c0258b0 tx=0x7f833c058f90).stop 2021-03-09T10:46:03.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:03.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=10831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:03.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:03.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:03.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:03.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324006e60 2021-03-09T10:46:03.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005040 con 0x7f8338071d10 2021-03-09T10:46:03.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024aa0 con 0x7f8324006600 2021-03-09T10:46:03.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:46:03.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:03.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:46:03.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:03.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:03.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:03.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.168+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:03.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.172+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:03.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.172+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:03.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.172+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028fa0 con 0x7f8338071d10 2021-03-09T10:46:03.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.172+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10839 cs=0 l=1 rev1=1 rx=0x7f833c05a880 tx=0x7f833c058f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:03.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.172+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338071d10 2021-03-09T10:46:03.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.172+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0218d0 con 0x7f8338071d10 2021-03-09T10:46:03.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:03.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c061300 con 0x7f8338071d10 2021-03-09T10:46:03.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:46:03 smithi146 bash[13136]: debug 2021-03-09T10:46:03.170+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31619 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:46:08.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:46:08.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:08.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10839 cs=0 l=1 rev1=1 rx=0x7f833c05a880 tx=0x7f833c058f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:08.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10839 cs=0 l=1 rev1=1 rx=0x7f833c05a880 tx=0x7f833c058f90).stop 2021-03-09T10:46:08.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:08.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=10839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:08.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:08.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324006e60 2021-03-09T10:46:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022270 con 0x7f83240069e0 2021-03-09T10:46:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a200 con 0x7f8324006600 2021-03-09T10:46:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:46:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:46:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:08.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:08.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:08.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:08.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064240 con 0x7f83240069e0 2021-03-09T10:46:08.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10848 cs=0 l=1 rev1=1 rx=0x7f833c054b20 tx=0x7f833c0a7000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:08.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa040 con 0x7f83240069e0 2021-03-09T10:46:08.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.180+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c054780 con 0x7f83240069e0 2021-03-09T10:46:08.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:08.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c010c30 con 0x7f83240069e0 2021-03-09T10:46:08.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:46:08 smithi146 bash[13136]: debug 2021-03-09T10:46:08.178+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31645 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:46:13.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:46:13.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:13.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10848 cs=0 l=1 rev1=1 rx=0x7f833c054b20 tx=0x7f833c0a7000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:13.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10848 cs=0 l=1 rev1=1 rx=0x7f833c054b20 tx=0x7f833c0a7000).stop 2021-03-09T10:46:13.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:13.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=10848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:13.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:13.182 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:13.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:13.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062720 con 0x7f8324006e60 2021-03-09T10:46:13.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8338071d10 2021-03-09T10:46:13.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801cd10 con 0x7f8324006600 2021-03-09T10:46:13.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:46:13.184 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:13.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:46:13.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:13.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.176+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:13.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.180+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:13.185 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.180+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:13.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.180+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:13.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.180+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:13.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.180+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e5c0 con 0x7f8338071d10 2021-03-09T10:46:13.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.180+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10856 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344004f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:13.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344075340 con 0x7f8338071d10 2021-03-09T10:46:13.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338071d10 2021-03-09T10:46:13.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:13.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405ec40 con 0x7f8338071d10 2021-03-09T10:46:13.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:46:13 smithi146 bash[13136]: debug 2021-03-09T10:46:13.182+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31672 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:46:18.186 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:46:18.187 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:18.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10856 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344004f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:18.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10856 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344004f90).stop 2021-03-09T10:46:18.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:18.188 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=10856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:18.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:18.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:18.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:18.189 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062720 con 0x7f8324006e60 2021-03-09T10:46:18.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ad70 con 0x7f83240069e0 2021-03-09T10:46:18.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324006600 2021-03-09T10:46:18.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:46:18.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:18.190 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:46:18.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:18.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.184+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:18.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.188+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:18.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.188+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:18.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.188+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:18.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.188+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:18.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.188+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380185f0 con 0x7f83240069e0 2021-03-09T10:46:18.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.188+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10867 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c0a7000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:18.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f83240069e0 2021-03-09T10:46:18.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c010890 con 0x7f83240069e0 2021-03-09T10:46:18.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:18.192+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02e350 con 0x7f83240069e0 2021-03-09T10:46:18.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:46:18 smithi146 bash[13136]: debug 2021-03-09T10:46:18.186+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31698 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:46:23.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:46:23.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:23.191 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10867 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c0a7000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:23.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10867 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c0a7000).stop 2021-03-09T10:46:23.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:23.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=10867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:23.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:23.192 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:23.193 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:23.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062080 con 0x7f8324006e60 2021-03-09T10:46:23.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062720 con 0x7f8338071d10 2021-03-09T10:46:23.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e920 con 0x7f8324006600 2021-03-09T10:46:23.194 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:46:23.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:23.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:46:23.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:23.195 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:23.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.188+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:23.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.192+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:23.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.192+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:23.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.192+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:23.196 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.192+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380185f0 con 0x7f8338071d10 2021-03-09T10:46:23.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.192+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10874 cs=0 l=1 rev1=1 rx=0x7f833c029870 tx=0x7f833c0a7000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:23.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.196+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338071d10 2021-03-09T10:46:23.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.196+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079630 con 0x7f8338071d10 2021-03-09T10:46:23.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:23.200+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c054310 con 0x7f8338071d10 2021-03-09T10:46:23.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:46:23 smithi146 bash[13136]: debug 2021-03-09T10:46:23.193+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31725 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:46:28.197 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.192+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:46:28.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.192+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:28.198 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.192+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10874 cs=0 l=1 rev1=1 rx=0x7f833c029870 tx=0x7f833c0a7000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:28.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.192+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10874 cs=0 l=1 rev1=1 rx=0x7f833c029870 tx=0x7f833c0a7000).stop 2021-03-09T10:46:28.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.192+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:28.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.192+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=10874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:28.199 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.192+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:28.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.192+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:28.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.196+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:28.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.196+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380670e0 con 0x7f8324006e60 2021-03-09T10:46:28.200 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.196+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024aa0 con 0x7f83240069e0 2021-03-09T10:46:28.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.196+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019270 con 0x7f8324006600 2021-03-09T10:46:28.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.196+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:46:28.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.196+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:28.201 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.196+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:46:28.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.196+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:28.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.196+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:28.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.196+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:28.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.196+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:28.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.196+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:28.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.196+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:28.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.196+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338064240 con 0x7f83240069e0 2021-03-09T10:46:28.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.196+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10882 cs=0 l=1 rev1=1 rx=0x7f8338020130 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:28.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.200+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f83240069e0 2021-03-09T10:46:28.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.200+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834403c280 con 0x7f83240069e0 2021-03-09T10:46:28.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:28.204+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834403cd20 con 0x7f83240069e0 2021-03-09T10:46:28.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:46:28 smithi146 bash[13136]: debug 2021-03-09T10:46:28.197+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31751 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:46:33.202 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:46:33.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:33.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10882 cs=0 l=1 rev1=1 rx=0x7f8338020130 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:33.203 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10882 cs=0 l=1 rev1=1 rx=0x7f8338020130 tx=0x7f8344004650).stop 2021-03-09T10:46:33.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:33.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=10882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:33.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:33.204 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:33.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:33.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324006e60 2021-03-09T10:46:33.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062080 con 0x7f8338071d10 2021-03-09T10:46:33.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d840 con 0x7f8324006600 2021-03-09T10:46:33.205 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:46:33.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:33.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:46:33.206 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:33.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:33.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:33.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:33.207 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:33.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:33.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801ab70 con 0x7f8338071d10 2021-03-09T10:46:33.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.204+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10892 cs=0 l=1 rev1=1 rx=0x7f833c012040 tx=0x7f833c0a7000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:33.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.208+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa080 con 0x7f8338071d10 2021-03-09T10:46:33.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.208+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c062a10 con 0x7f8338071d10 2021-03-09T10:46:33.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:33.212+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0792c0 con 0x7f8338071d10 2021-03-09T10:46:33.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:46:33 smithi146 bash[13136]: debug 2021-03-09T10:46:33.205+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31778 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:46:38.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.204+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:46:38.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.204+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:38.208 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.204+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10892 cs=0 l=1 rev1=1 rx=0x7f833c012040 tx=0x7f833c0a7000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:38.209 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.204+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10892 cs=0 l=1 rev1=1 rx=0x7f833c012040 tx=0x7f833c0a7000).stop 2021-03-09T10:46:38.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.204+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:38.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.204+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=10892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:38.210 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.204+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:38.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.204+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:38.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.204+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:38.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.204+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c9e0 con 0x7f8324006e60 2021-03-09T10:46:38.211 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.204+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063460 con 0x7f83240069e0 2021-03-09T10:46:38.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.204+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062720 con 0x7f8324006600 2021-03-09T10:46:38.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.204+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:46:38.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.204+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:38.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.204+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:46:38.212 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.204+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:38.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.208+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:38.213 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.208+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:38.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.208+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:38.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.208+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:38.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.208+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:38.214 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.208+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b160 con 0x7f83240069e0 2021-03-09T10:46:38.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.208+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10900 cs=0 l=1 rev1=1 rx=0x7f834400e240 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:38.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.212+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344066980 con 0x7f83240069e0 2021-03-09T10:46:38.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.212+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f83240069e0 2021-03-09T10:46:38.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:38.216+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344049710 con 0x7f83240069e0 2021-03-09T10:46:38.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:46:38 smithi146 bash[13136]: debug 2021-03-09T10:46:38.209+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31804 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:46:43.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:46:43.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:43.215 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10900 cs=0 l=1 rev1=1 rx=0x7f834400e240 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:43.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10900 cs=0 l=1 rev1=1 rx=0x7f834400e240 tx=0x7f8344004650).stop 2021-03-09T10:46:43.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:43.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=10900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:43.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:43.216 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:43.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:43.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062080 con 0x7f8324006e60 2021-03-09T10:46:43.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065d20 con 0x7f8338071d10 2021-03-09T10:46:43.217 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062720 con 0x7f8324006600 2021-03-09T10:46:43.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:46:43.218 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:43.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:46:43.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:43.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:43.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:43.219 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:43.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:43.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:43.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ba00 con 0x7f8338071d10 2021-03-09T10:46:43.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.216+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028550 secure :-1 s=READY pgs=10909 cs=0 l=1 rev1=1 rx=0x7f833c06b510 tx=0x7f833c0a7000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:43.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.220+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa030 con 0x7f8338071d10 2021-03-09T10:46:43.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.220+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079630 con 0x7f8338071d10 2021-03-09T10:46:43.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:43.220+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c045000 con 0x7f8338071d10 2021-03-09T10:46:43.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:46:43 smithi146 bash[13136]: debug 2021-03-09T10:46:43.217+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31831 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:46:48.220 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.215+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338028550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:46:48.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.215+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338028550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:48.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.215+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028550 secure :-1 s=READY pgs=10909 cs=0 l=1 rev1=1 rx=0x7f833c06b510 tx=0x7f833c0a7000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:48.221 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.215+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028550 secure :-1 s=READY pgs=10909 cs=0 l=1 rev1=1 rx=0x7f833c06b510 tx=0x7f833c0a7000).stop 2021-03-09T10:46:48.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338028550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:48.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.215+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338028550 unknown :-1 s=CLOSED pgs=10909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:48.222 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.215+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:48.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.215+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:48.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.215+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:48.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380241c0 con 0x7f8324006e60 2021-03-09T10:46:48.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062720 con 0x7f83240069e0 2021-03-09T10:46:48.223 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.215+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380729d0 con 0x7f8324006600 2021-03-09T10:46:48.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.215+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:46:48.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.215+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:48.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.219+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:46:48.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.219+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:48.224 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:48.225 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.219+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:48.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:48.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.219+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:48.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:48.226 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.219+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065090 con 0x7f83240069e0 2021-03-09T10:46:48.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.219+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10917 cs=0 l=1 rev1=1 rx=0x7f833c01da80 tx=0x7f833c0a7000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:48.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa040 con 0x7f83240069e0 2021-03-09T10:46:48.228 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.223+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f83240069e0 2021-03-09T10:46:48.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:48.227+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f83240069e0 2021-03-09T10:46:48.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:46:48 smithi146 bash[13136]: debug 2021-03-09T10:46:48.221+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31857 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:46:53.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:46:53.227 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10917 cs=0 l=1 rev1=1 rx=0x7f833c01da80 tx=0x7f833c0a7000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10917 cs=0 l=1 rev1=1 rx=0x7f833c01da80 tx=0x7f833c0a7000).stop 2021-03-09T10:46:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=10917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:53.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:53.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:53.230 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a580 con 0x7f8324006e60 2021-03-09T10:46:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065d20 con 0x7f8338071d10 2021-03-09T10:46:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005840 con 0x7f8324006600 2021-03-09T10:46:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:46:53.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:46:53.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:53.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:53.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.227+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:53.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.231+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:53.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.231+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:53.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.231+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:53.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.231+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:53.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.231+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380185f0 con 0x7f8338071d10 2021-03-09T10:46:53.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.231+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10926 cs=0 l=1 rev1=1 rx=0x7f833c01c4a0 tx=0x7f833c0a7000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:53.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.231+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c041930 con 0x7f8338071d10 2021-03-09T10:46:53.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.231+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c041750 con 0x7f8338071d10 2021-03-09T10:46:53.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:53.235+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0792c0 con 0x7f8338071d10 2021-03-09T10:46:53.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:46:53 smithi146 bash[13136]: debug 2021-03-09T10:46:53.229+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31883 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:46:58.232 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.231+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:46:58.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.231+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:58.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.231+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10926 cs=0 l=1 rev1=1 rx=0x7f833c01c4a0 tx=0x7f833c0a7000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:58.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.231+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10926 cs=0 l=1 rev1=1 rx=0x7f833c01c4a0 tx=0x7f833c0a7000).stop 2021-03-09T10:46:58.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.231+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:58.233 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.231+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=10926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:58.234 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.231+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:58.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.231+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:58.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.231+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:58.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.231+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324006e60 2021-03-09T10:46:58.235 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.231+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f83240069e0 2021-03-09T10:46:58.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.231+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062080 con 0x7f8324006600 2021-03-09T10:46:58.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.231+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:46:58.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.231+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:58.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.235+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:46:58.236 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.235+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:58.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.235+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:58.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.235+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:58.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.235+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:58.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.235+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:58.237 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.235+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:58.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.235+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b160 con 0x7f83240069e0 2021-03-09T10:46:58.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.235+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10935 cs=0 l=1 rev1=1 rx=0x7f8338071ce0 tx=0x7f833c0a7000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:58.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.239+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01de20 con 0x7f83240069e0 2021-03-09T10:46:58.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.239+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0933d0 con 0x7f83240069e0 2021-03-09T10:46:58.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:46:58.243+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0792c0 con 0x7f83240069e0 2021-03-09T10:46:58.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:46:58 smithi146 bash[13136]: debug 2021-03-09T10:46:58.233+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31910 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:47:03.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:47:03.238 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:03.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10935 cs=0 l=1 rev1=1 rx=0x7f8338071ce0 tx=0x7f833c0a7000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:03.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10935 cs=0 l=1 rev1=1 rx=0x7f8338071ce0 tx=0x7f833c0a7000).stop 2021-03-09T10:47:03.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:03.239 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=10935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:03.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:03.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:03.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:03.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380241c0 con 0x7f8324006e60 2021-03-09T10:47:03.240 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380729d0 con 0x7f8338071d10 2021-03-09T10:47:03.241 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a50 con 0x7f8324006600 2021-03-09T10:47:03.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:47:03.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:03.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:47:03.242 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:03.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:03.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:03.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:03.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:03.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:03.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.239+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380185f0 con 0x7f8338071d10 2021-03-09T10:47:03.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.243+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10945 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:03.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.243+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f8338071d10 2021-03-09T10:47:03.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.243+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405ec40 con 0x7f8338071d10 2021-03-09T10:47:03.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:03.247+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338071d10 2021-03-09T10:47:03.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:47:03 smithi146 bash[13136]: debug 2021-03-09T10:47:03.237+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31936 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:47:08.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:47:08.243 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:08.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10945 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:08.244 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10945 cs=0 l=1 rev1=1 rx=0x7f834400bf70 tx=0x7f8344004650).stop 2021-03-09T10:47:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=10945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065d20 con 0x7f8324006e60 2021-03-09T10:47:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060b00 con 0x7f83240069e0 2021-03-09T10:47:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324006600 2021-03-09T10:47:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:47:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:47:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.243+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:08.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.247+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:08.248 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.247+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.247+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.247+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.247+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800b360 con 0x7f83240069e0 2021-03-09T10:47:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.247+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10952 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c0a7000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.251+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f83240069e0 2021-03-09T10:47:08.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.251+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f83240069e0 2021-03-09T10:47:08.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:08.255+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c062cb0 con 0x7f83240069e0 2021-03-09T10:47:08.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:47:08 smithi146 bash[13136]: debug 2021-03-09T10:47:08.245+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31963 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:47:13.250 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:47:13.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:13.251 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10952 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c0a7000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:13.252 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 secure :-1 s=READY pgs=10952 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c0a7000).stop 2021-03-09T10:47:13.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:13.253 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83240069e0 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=10952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:13.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:13.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:13.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:13.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062080 con 0x7f8324006e60 2021-03-09T10:47:13.254 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062720 con 0x7f8338071d10 2021-03-09T10:47:13.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380241c0 con 0x7f8324006600 2021-03-09T10:47:13.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:47:13.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:13.255 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:47:13.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:13.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:13.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:13.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.251+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:13.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.255+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:13.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.255+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:13.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.255+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338008380 con 0x7f8338071d10 2021-03-09T10:47:13.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.255+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10960 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:13.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.259+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405bea0 con 0x7f8338071d10 2021-03-09T10:47:13.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.259+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405bea0 con 0x7f8338071d10 2021-03-09T10:47:13.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:13.263+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338071d10 2021-03-09T10:47:13.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:47:13 smithi146 bash[13136]: debug 2021-03-09T10:47:13.253+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 31989 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:47:18.256 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.255+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:47:18.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.255+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:18.257 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.255+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10960 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:18.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.255+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10960 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344004650).stop 2021-03-09T10:47:18.258 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.255+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:18.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.255+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=10960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:18.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.255+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:18.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.255+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:18.259 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.255+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:18.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.255+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801cd10 con 0x7f8324006e60 2021-03-09T10:47:18.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.255+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c9a0 con 0x7f8338069320 2021-03-09T10:47:18.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.255+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380241c0 con 0x7f8324006600 2021-03-09T10:47:18.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.255+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:47:18.260 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.255+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:18.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.259+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:47:18.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.259+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:18.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.259+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:18.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.259+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:18.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.259+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:18.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.259+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:18.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.259+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:18.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.259+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806b160 con 0x7f8338069320 2021-03-09T10:47:18.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.259+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=10970 cs=0 l=1 rev1=1 rx=0x7f8338065060 tx=0x7f8344014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:18.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.263+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403bdf0 con 0x7f8338069320 2021-03-09T10:47:18.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.263+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405f830 con 0x7f8338069320 2021-03-09T10:47:18.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:18.267+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405bd70 con 0x7f8338069320 2021-03-09T10:47:18.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:47:18 smithi146 bash[13136]: debug 2021-03-09T10:47:18.257+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32016 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:47:23.261 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:47:23.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:23.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=10970 cs=0 l=1 rev1=1 rx=0x7f8338065060 tx=0x7f8344014000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:23.262 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=10970 cs=0 l=1 rev1=1 rx=0x7f8338065060 tx=0x7f8344014000).stop 2021-03-09T10:47:23.263 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:23.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=10970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:23.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:23.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:23.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:23.264 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324006e60 2021-03-09T10:47:23.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8338071d10 2021-03-09T10:47:23.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063460 con 0x7f8324006600 2021-03-09T10:47:23.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:47:23.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:23.265 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:47:23.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:23.266 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.263+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:23.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.267+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:23.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.267+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:23.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.267+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:23.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.267+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:23.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.267+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801fb50 con 0x7f8338071d10 2021-03-09T10:47:23.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.267+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10978 cs=0 l=1 rev1=1 rx=0x7f833c0451a0 tx=0x7f833c0a7000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:23.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.271+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c045000 con 0x7f8338071d10 2021-03-09T10:47:23.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.271+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079800 con 0x7f8338071d10 2021-03-09T10:47:23.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:23.271+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f8338071d10 2021-03-09T10:47:23.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:47:23 smithi146 bash[13136]: debug 2021-03-09T10:47:23.261+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32041 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:47:28.267 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:47:28.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:28.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10978 cs=0 l=1 rev1=1 rx=0x7f833c0451a0 tx=0x7f833c0a7000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:28.268 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10978 cs=0 l=1 rev1=1 rx=0x7f833c0451a0 tx=0x7f833c0a7000).stop 2021-03-09T10:47:28.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:28.269 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=10978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:28.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:28.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:28.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:28.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006e60 2021-03-09T10:47:28.270 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062720 con 0x7f8338069320 2021-03-09T10:47:28.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062080 con 0x7f8324006600 2021-03-09T10:47:28.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:47:28.271 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:28.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:47:28.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:28.272 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.267+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:28.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.271+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:28.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.271+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:28.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.271+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:28.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.271+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:28.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.271+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338071b20 con 0x7f8338069320 2021-03-09T10:47:28.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.271+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=10986 cs=0 l=1 rev1=1 rx=0x7f833c04a470 tx=0x7f833c0796a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:28.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.275+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338069320 2021-03-09T10:47:28.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.275+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0070d0 con 0x7f8338069320 2021-03-09T10:47:28.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:28.279+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338069320 2021-03-09T10:47:28.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:47:28 smithi146 bash[13136]: debug 2021-03-09T10:47:28.268+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32069 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:47:33.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:47:33.273 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:33.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=10986 cs=0 l=1 rev1=1 rx=0x7f833c04a470 tx=0x7f833c0796a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:33.274 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=10986 cs=0 l=1 rev1=1 rx=0x7f833c04a470 tx=0x7f833c0796a0).stop 2021-03-09T10:47:33.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:33.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=10986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:33.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:33.275 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:33.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:33.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380241c0 con 0x7f8324006e60 2021-03-09T10:47:33.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8338071d10 2021-03-09T10:47:33.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006600 2021-03-09T10:47:33.276 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:47:33.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:33.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:47:33.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:33.277 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:33.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:33.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:33.278 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:33.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:33.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.274+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ba00 con 0x7f8338071d10 2021-03-09T10:47:33.279 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.278+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10996 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c0796a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:33.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.282+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338071d10 2021-03-09T10:47:33.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.282+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c040240 con 0x7f8338071d10 2021-03-09T10:47:33.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:33.286+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c093880 con 0x7f8338071d10 2021-03-09T10:47:33.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:47:33 smithi146 bash[13136]: debug 2021-03-09T10:47:33.272+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32094 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:47:38.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.278+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:47:38.280 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.278+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:38.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.278+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10996 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c0796a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:38.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.278+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=10996 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c0796a0).stop 2021-03-09T10:47:38.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.278+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:38.281 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.278+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=10996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:38.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.278+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:38.282 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.278+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:38.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.278+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:38.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.278+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062720 con 0x7f8324006e60 2021-03-09T10:47:38.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.278+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8338069320 2021-03-09T10:47:38.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.278+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062080 con 0x7f8324006600 2021-03-09T10:47:38.283 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.278+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:47:38.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.278+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:38.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.278+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:47:38.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.278+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:38.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.282+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:38.284 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.282+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:38.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.282+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:38.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.282+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:38.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.282+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:38.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.282+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380735b0 con 0x7f8338069320 2021-03-09T10:47:38.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.282+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11004 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c0796a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:38.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.286+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f8338069320 2021-03-09T10:47:38.287 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.286+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f8338069320 2021-03-09T10:47:38.288 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:38.286+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c010520 con 0x7f8338069320 2021-03-09T10:47:38.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:47:38 smithi146 bash[13136]: debug 2021-03-09T10:47:38.280+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32122 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:47:43.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:47:43.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:43.285 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11004 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c0796a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:43.286 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11004 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c0796a0).stop 2021-03-09T10:47:43.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:43.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=11004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:43.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8324006e60 2021-03-09T10:47:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8338071d10 2021-03-09T10:47:43.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e1d0 con 0x7f8324006600 2021-03-09T10:47:43.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:47:43.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:43.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:47:43.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:43.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:43.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:43.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:43.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:43.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:43.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801ccc0 con 0x7f8338071d10 2021-03-09T10:47:43.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.286+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11012 cs=0 l=1 rev1=1 rx=0x7f833c025e70 tx=0x7f833c0796a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:43.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.290+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c025cb0 con 0x7f8338071d10 2021-03-09T10:47:43.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.290+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f8338071d10 2021-03-09T10:47:43.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:43.294+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f8338071d10 2021-03-09T10:47:43.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:47:43 smithi146 bash[13136]: debug 2021-03-09T10:47:43.284+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32147 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:47:48.290 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:47:48.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:48.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11012 cs=0 l=1 rev1=1 rx=0x7f833c025e70 tx=0x7f833c0796a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:48.291 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11012 cs=0 l=1 rev1=1 rx=0x7f833c025e70 tx=0x7f833c0796a0).stop 2021-03-09T10:47:48.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:48.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:48.292 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:48.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:48.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:48.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074780 con 0x7f8324006e60 2021-03-09T10:47:48.293 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016e40 con 0x7f8338069320 2021-03-09T10:47:48.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011800 con 0x7f8324006600 2021-03-09T10:47:48.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:47:48.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:48.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:47:48.294 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:48.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.290+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:48.295 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.294+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:48.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.294+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:48.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.294+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:48.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.294+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:48.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.294+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800b360 con 0x7f8338069320 2021-03-09T10:47:48.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.294+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11021 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c079630).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:48.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.298+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa030 con 0x7f8338069320 2021-03-09T10:47:48.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.298+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c02eec0 con 0x7f8338069320 2021-03-09T10:47:48.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:48.302+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c048ca0 con 0x7f8338069320 2021-03-09T10:47:48.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:47:48 smithi146 bash[13136]: debug 2021-03-09T10:47:48.288+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32175 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:47:53.296 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:47:53.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:53.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11021 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c079630).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:53.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11021 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c079630).stop 2021-03-09T10:47:53.297 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:53.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=11021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:53.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:53.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:53.299 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:53.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006e60 2021-03-09T10:47:53.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062720 con 0x7f8338071d10 2021-03-09T10:47:53.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062080 con 0x7f8324006600 2021-03-09T10:47:53.300 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:47:53.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:53.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:47:53.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:53.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.294+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:53.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.298+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:53.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.298+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:53.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.298+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:53.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.298+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:53.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.298+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024170 con 0x7f8338071d10 2021-03-09T10:47:53.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.298+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11030 cs=0 l=1 rev1=1 rx=0x7f833c062f40 tx=0x7f833c0796a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:53.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.302+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338071d10 2021-03-09T10:47:53.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.302+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0576a0 con 0x7f8338071d10 2021-03-09T10:47:53.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:53.302+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c2a0 con 0x7f8338071d10 2021-03-09T10:47:53.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:47:53 smithi146 bash[13136]: debug 2021-03-09T10:47:53.296+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32200 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:47:58.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.298+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:47:58.301 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.298+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:58.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.298+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11030 cs=0 l=1 rev1=1 rx=0x7f833c062f40 tx=0x7f833c0796a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:58.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.298+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11030 cs=0 l=1 rev1=1 rx=0x7f833c062f40 tx=0x7f833c0796a0).stop 2021-03-09T10:47:58.302 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:58.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:58.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:58.303 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:58.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:47:58.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:58.304 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:58.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:58.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e1d0 con 0x7f8324006e60 2021-03-09T10:47:58.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8338069320 2021-03-09T10:47:58.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a50 con 0x7f8324006600 2021-03-09T10:47:58.305 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:47:58.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:58.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:58.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:58.306 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:58.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:58.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.302+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024170 con 0x7f8338069320 2021-03-09T10:47:58.307 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.306+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11038 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c0796a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.306+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa030 con 0x7f8338069320 2021-03-09T10:47:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.306+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c048900 con 0x7f8338069320 2021-03-09T10:47:58.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:47:58.310+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0648d0 con 0x7f8338069320 2021-03-09T10:47:58.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:47:58 smithi146 bash[13136]: debug 2021-03-09T10:47:58.300+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32228 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:48:03.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.306+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:48:03.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.306+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:03.311 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.306+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11038 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c0796a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:03.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.306+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11038 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c0796a0).stop 2021-03-09T10:48:03.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.306+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:03.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.306+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=11038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:03.312 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.306+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:03.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.306+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:03.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.306+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:48:03.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.306+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:03.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.306+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:03.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.306+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324006e60 2021-03-09T10:48:03.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.306+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8338071d10 2021-03-09T10:48:03.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.306+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062720 con 0x7f8324006600 2021-03-09T10:48:03.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.306+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:48:03.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.306+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:03.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.310+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:03.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.310+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:03.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.310+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:03.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.310+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:03.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.310+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:03.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.310+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380641e0 con 0x7f8338071d10 2021-03-09T10:48:03.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.310+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11047 cs=0 l=1 rev1=1 rx=0x7f833c0645d0 tx=0x7f833c0796a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:03.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.314+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338071d10 2021-03-09T10:48:03.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.314+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0428c0 con 0x7f8338071d10 2021-03-09T10:48:03.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:03.314+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00eac0 con 0x7f8338071d10 2021-03-09T10:48:03.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:48:03 smithi146 bash[13136]: debug 2021-03-09T10:48:03.308+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32253 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:48:08.313 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.310+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:48:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.310+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.310+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11047 cs=0 l=1 rev1=1 rx=0x7f833c0645d0 tx=0x7f833c0796a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.310+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11047 cs=0 l=1 rev1=1 rx=0x7f833c0645d0 tx=0x7f833c0796a0).stop 2021-03-09T10:48:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.310+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.310+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.310+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.310+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.310+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.310+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006e60 2021-03-09T10:48:08.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.310+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8338069320 2021-03-09T10:48:08.316 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.310+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e1d0 con 0x7f8324006600 2021-03-09T10:48:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.314+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:48:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.314+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.314+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:48:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.314+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:08.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.314+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:08.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.314+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:08.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.314+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:08.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.314+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:08.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.314+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:08.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.314+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024170 con 0x7f8338069320 2021-03-09T10:48:08.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.314+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11057 cs=0 l=1 rev1=1 rx=0x7f833c029620 tx=0x7f833c0796a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:08.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.318+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa020 con 0x7f8338069320 2021-03-09T10:48:08.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.318+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c029690 con 0x7f8338069320 2021-03-09T10:48:08.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:08.322+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f8338069320 2021-03-09T10:48:08.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:48:08 smithi146 bash[13136]: debug 2021-03-09T10:48:08.312+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32281 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:48:13.318 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:48:13.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:13.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11057 cs=0 l=1 rev1=1 rx=0x7f833c029620 tx=0x7f833c0796a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:13.319 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11057 cs=0 l=1 rev1=1 rx=0x7f833c029620 tx=0x7f833c0796a0).stop 2021-03-09T10:48:13.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:13.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=11057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:13.320 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338064fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:13.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:13.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:48:13.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:13.321 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:13.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012720 con 0x7f8324006e60 2021-03-09T10:48:13.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8338071d10 2021-03-09T10:48:13.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006600 2021-03-09T10:48:13.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338064fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:13.322 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:48:13.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:13.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:13.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:13.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:13.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:13.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.318+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800d8f0 con 0x7f8338071d10 2021-03-09T10:48:13.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.322+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338064fa0 secure :-1 s=READY pgs=11064 cs=0 l=1 rev1=1 rx=0x7f833c01a610 tx=0x7f833c0796a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:13.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.326+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0294d0 con 0x7f8338071d10 2021-03-09T10:48:13.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.326+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c05abf0 con 0x7f8338071d10 2021-03-09T10:48:13.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:13.326+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02fc60 con 0x7f8338071d10 2021-03-09T10:48:13.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:48:13 smithi146 bash[13136]: debug 2021-03-09T10:48:13.316+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32306 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:48:18.323 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338064fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:48:18.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338064fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:18.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338064fa0 secure :-1 s=READY pgs=11064 cs=0 l=1 rev1=1 rx=0x7f833c01a610 tx=0x7f833c0796a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:18.324 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338064fa0 secure :-1 s=READY pgs=11064 cs=0 l=1 rev1=1 rx=0x7f833c01a610 tx=0x7f833c0796a0).stop 2021-03-09T10:48:18.325 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338064fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:18.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338064fa0 unknown :-1 s=CLOSED pgs=11064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:18.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:18.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:18.326 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:18.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062720 con 0x7f8324006e60 2021-03-09T10:48:18.327 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002ea0 con 0x7f8338069320 2021-03-09T10:48:18.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e1d0 con 0x7f8324006600 2021-03-09T10:48:18.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:48:18.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:18.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:48:18.328 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:18.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.322+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:18.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.326+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:18.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.326+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:18.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.326+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:18.329 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.326+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:18.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.326+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028fa0 con 0x7f8338069320 2021-03-09T10:48:18.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.326+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11073 cs=0 l=1 rev1=1 rx=0x7f8344043f50 tx=0x7f8344014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:18.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.330+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:48:18.331 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.330+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338069320 2021-03-09T10:48:18.334 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:18.334+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405bd70 con 0x7f8338069320 2021-03-09T10:48:18.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:48:18 smithi146 bash[13136]: debug 2021-03-09T10:48:18.328+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32330 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:48:23.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:48:23.330 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:23.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11073 cs=0 l=1 rev1=1 rx=0x7f8344043f50 tx=0x7f8344014000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:23.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11073 cs=0 l=1 rev1=1 rx=0x7f8344043f50 tx=0x7f8344014000).stop 2021-03-09T10:48:23.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:23.332 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=11073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:23.334 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:23.334 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:23.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:23.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338002d60 con 0x7f8324006e60 2021-03-09T10:48:23.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8338071d10 2021-03-09T10:48:23.335 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006600 2021-03-09T10:48:23.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:48:23.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:23.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:48:23.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:23.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.329+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:23.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.333+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:23.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.333+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:23.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.333+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:23.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.333+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:23.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.333+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e6f0 con 0x7f8338071d10 2021-03-09T10:48:23.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.333+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11083 cs=0 l=1 rev1=1 rx=0x7f834402b760 tx=0x7f8344014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:23.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.333+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402ba00 con 0x7f8338071d10 2021-03-09T10:48:23.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.337+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404aa60 con 0x7f8338071d10 2021-03-09T10:48:23.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:23.337+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402b220 con 0x7f8338071d10 2021-03-09T10:48:23.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:48:23 smithi146 bash[13136]: debug 2021-03-09T10:48:23.332+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32354 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:48:28.336 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.333+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:48:28.337 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.333+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:28.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.333+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11083 cs=0 l=1 rev1=1 rx=0x7f834402b760 tx=0x7f8344014000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:28.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.333+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11083 cs=0 l=1 rev1=1 rx=0x7f834402b760 tx=0x7f8344014000).stop 2021-03-09T10:48:28.338 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.333+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:28.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.333+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:28.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.333+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:28.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.333+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:28.339 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.333+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:28.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.333+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e1d0 con 0x7f8324006e60 2021-03-09T10:48:28.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.333+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8338069320 2021-03-09T10:48:28.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.333+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8324006600 2021-03-09T10:48:28.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.333+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:48:28.340 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.333+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:28.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.333+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:28.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.337+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:48:28.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.337+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:28.341 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.337+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:28.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.337+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:28.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.337+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:28.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.337+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:28.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.337+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338065eb0 con 0x7f8338069320 2021-03-09T10:48:28.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.337+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11090 cs=0 l=1 rev1=1 rx=0x7f833c01c6a0 tx=0x7f833c079930).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:28.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.341+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f8338069320 2021-03-09T10:48:28.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.341+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c048900 con 0x7f8338069320 2021-03-09T10:48:28.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:28.345+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338069320 2021-03-09T10:48:28.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:48:28 smithi146 bash[13136]: debug 2021-03-09T10:48:28.340+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32382 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:48:33.342 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:48:33.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:33.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11090 cs=0 l=1 rev1=1 rx=0x7f833c01c6a0 tx=0x7f833c079930).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:33.343 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11090 cs=0 l=1 rev1=1 rx=0x7f833c01c6a0 tx=0x7f833c079930).stop 2021-03-09T10:48:33.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:33.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=11090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:33.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:33.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:33.344 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:33.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006e60 2021-03-09T10:48:33.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023ce0 con 0x7f8338071d10 2021-03-09T10:48:33.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8324006600 2021-03-09T10:48:33.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:48:33.345 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:33.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:48:33.346 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:33.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:33.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:33.347 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:33.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:33.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:33.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.341+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020c20 con 0x7f8338071d10 2021-03-09T10:48:33.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.345+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11100 cs=0 l=1 rev1=1 rx=0x7f833c05efd0 tx=0x7f833c079930).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:33.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.345+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338071d10 2021-03-09T10:48:33.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c048900 con 0x7f8338071d10 2021-03-09T10:48:33.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:33.349+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01d3a0 con 0x7f8338071d10 2021-03-09T10:48:33.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:48:33 smithi146 bash[13136]: debug 2021-03-09T10:48:33.343+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32407 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:48:38.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.345+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:48:38.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.345+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:38.348 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.345+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11100 cs=0 l=1 rev1=1 rx=0x7f833c05efd0 tx=0x7f833c079930).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:38.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.345+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11100 cs=0 l=1 rev1=1 rx=0x7f833c05efd0 tx=0x7f833c079930).stop 2021-03-09T10:48:38.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.345+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:38.349 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.345+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:38.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.345+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:38.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.345+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:38.350 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.345+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:38.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.345+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e880 con 0x7f8324006e60 2021-03-09T10:48:38.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.345+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064990 con 0x7f8338069320 2021-03-09T10:48:38.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.345+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018af0 con 0x7f8324006600 2021-03-09T10:48:38.351 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.345+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:48:38.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.345+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:38.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.345+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:48:38.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.345+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:38.352 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.349+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:38.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.349+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:38.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.349+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:38.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.349+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:38.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.349+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:38.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.349+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338015b00 con 0x7f8338069320 2021-03-09T10:48:38.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.349+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11110 cs=0 l=1 rev1=1 rx=0x7f834404acb0 tx=0x7f8344014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:38.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402b7d0 con 0x7f8338069320 2021-03-09T10:48:38.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402c800 con 0x7f8338069320 2021-03-09T10:48:38.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:38.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405bd70 con 0x7f8338069320 2021-03-09T10:48:38.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:48:38 smithi146 bash[13136]: debug 2021-03-09T10:48:38.351+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32435 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:48:43.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:48:43.353 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:43.354 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11110 cs=0 l=1 rev1=1 rx=0x7f834404acb0 tx=0x7f8344014000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:43.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11110 cs=0 l=1 rev1=1 rx=0x7f834404acb0 tx=0x7f8344014000).stop 2021-03-09T10:48:43.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:43.355 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=11110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:43.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:43.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:43.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:43.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019270 con 0x7f8324006e60 2021-03-09T10:48:43.356 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8338071d10 2021-03-09T10:48:43.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006600 2021-03-09T10:48:43.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:48:43.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:48:43.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:43.357 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:43.358 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:43.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:43.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:43.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:43.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:43.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.353+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338002b70 con 0x7f8338071d10 2021-03-09T10:48:43.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.357+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11118 cs=0 l=1 rev1=1 rx=0x7f833c01e040 tx=0x7f833c079930).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:43.360 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338071d10 2021-03-09T10:48:43.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c02f3b0 con 0x7f8338071d10 2021-03-09T10:48:43.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:43.361+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338071d10 2021-03-09T10:48:43.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:48:43 smithi146 bash[13136]: debug 2021-03-09T10:48:43.355+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32460 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:48:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:48:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11118 cs=0 l=1 rev1=1 rx=0x7f833c01e040 tx=0x7f833c079930).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11118 cs=0 l=1 rev1=1 rx=0x7f833c01e040 tx=0x7f833c079930).stop 2021-03-09T10:48:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800e1d0 con 0x7f8324006e60 2021-03-09T10:48:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f8338069320 2021-03-09T10:48:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f8324006600 2021-03-09T10:48:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:48:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:48:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:48.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:48.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:48.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:48.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.357+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017080 con 0x7f8338069320 2021-03-09T10:48:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.361+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11125 cs=0 l=1 rev1=1 rx=0x7f833c093570 tx=0x7f833c079930).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c007640 con 0x7f8338069320 2021-03-09T10:48:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c048900 con 0x7f8338069320 2021-03-09T10:48:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:48.369+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c093d90 con 0x7f8338069320 2021-03-09T10:48:48.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:48:48 smithi146 bash[13136]: debug 2021-03-09T10:48:48.363+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32487 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:48:53.364 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:48:53.365 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11125 cs=0 l=1 rev1=1 rx=0x7f833c093570 tx=0x7f833c079930).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 secure :-1 s=READY pgs=11125 cs=0 l=1 rev1=1 rx=0x7f833c093570 tx=0x7f833c079930).stop 2021-03-09T10:48:53.366 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 msgr2=0x7f833802b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:53.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338069320 0x7f833802b2d0 unknown :-1 s=CLOSED pgs=11125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:53.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:53.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:53.367 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:53.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018af0 con 0x7f8324006e60 2021-03-09T10:48:53.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8338071d10 2021-03-09T10:48:53.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006600 2021-03-09T10:48:53.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:48:53.368 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:53.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:48:53.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:53.369 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:53.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:53.370 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:53.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:53.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:53.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017be0 con 0x7f8338071d10 2021-03-09T10:48:53.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.365+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11135 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c079930).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:53.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.369+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0252e0 con 0x7f8338071d10 2021-03-09T10:48:53.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.369+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c048900 con 0x7f8338071d10 2021-03-09T10:48:53.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:53.373+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c093c80 con 0x7f8338071d10 2021-03-09T10:48:53.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:48:53 smithi146 bash[13136]: debug 2021-03-09T10:48:53.367+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32513 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:48:58.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:48:58.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:58.371 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11135 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c079930).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:58.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11135 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c079930).stop 2021-03-09T10:48:58.372 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:58.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:58.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:58.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:58.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:58.373 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006e60 2021-03-09T10:48:58.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f83380292c0 2021-03-09T10:48:58.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800cce0 con 0x7f8324006600 2021-03-09T10:48:58.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:58.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:58.374 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:58.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:48:58.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:58.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:58.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:58.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.369+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802d440 con 0x7f83380292c0 2021-03-09T10:48:58.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.373+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 secure :-1 s=READY pgs=11143 cs=0 l=1 rev1=1 rx=0x7f834404b530 tx=0x7f8344014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:58.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.373+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344003d80 con 0x7f83380292c0 2021-03-09T10:48:58.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.373+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344020170 con 0x7f83380292c0 2021-03-09T10:48:58.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:48:58.377+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344049740 con 0x7f83380292c0 2021-03-09T10:48:58.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:48:58 smithi146 bash[13136]: debug 2021-03-09T10:48:58.371+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32540 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:49:03.375 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338077130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:49:03.376 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338077130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:03.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 secure :-1 s=READY pgs=11143 cs=0 l=1 rev1=1 rx=0x7f834404b530 tx=0x7f8344014000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:03.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 secure :-1 s=READY pgs=11143 cs=0 l=1 rev1=1 rx=0x7f834404b530 tx=0x7f8344014000).stop 2021-03-09T10:49:03.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338077130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:03.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 unknown :-1 s=CLOSED pgs=11143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:03.377 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:03.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:03.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:03.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324006e60 2021-03-09T10:49:03.378 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e740 con 0x7f8338071d10 2021-03-09T10:49:03.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324006600 2021-03-09T10:49:03.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:49:03.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:03.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:49:03.379 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:03.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:03.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:03.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:03.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:03.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:03.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338019f60 con 0x7f8338071d10 2021-03-09T10:49:03.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.377+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11151 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:03.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.381+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440470d0 con 0x7f8338071d10 2021-03-09T10:49:03.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.381+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344047230 con 0x7f8338071d10 2021-03-09T10:49:03.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:03.385+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344047080 con 0x7f8338071d10 2021-03-09T10:49:03.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:49:03 smithi146 bash[13136]: debug 2021-03-09T10:49:03.379+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32566 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:49:08.381 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:49:08.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:08.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11151 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344014000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:08.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11151 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344014000).stop 2021-03-09T10:49:08.382 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:08.383 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:08.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:08.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:49:08.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:08.384 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338016110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006e60 2021-03-09T10:49:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800cce0 con 0x7f83380292c0 2021-03-09T10:49:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338018af0 con 0x7f8324006600 2021-03-09T10:49:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:49:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338016110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338069170 con 0x7f83380292c0 2021-03-09T10:49:08.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.380+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338016110 secure :-1 s=READY pgs=11160 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:08.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.384+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344010ef0 con 0x7f83380292c0 2021-03-09T10:49:08.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405d550 con 0x7f83380292c0 2021-03-09T10:49:08.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:08.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834403cd40 con 0x7f83380292c0 2021-03-09T10:49:08.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:49:08 smithi146 bash[13136]: debug 2021-03-09T10:49:08.383+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32589 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:49:13.387 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338016110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:49:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338016110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338016110 secure :-1 s=READY pgs=11160 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344014000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338016110 secure :-1 s=READY pgs=11160 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344014000).stop 2021-03-09T10:49:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338016110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:13.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338016110 unknown :-1 s=CLOSED pgs=11160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:13.389 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324006e60 2021-03-09T10:49:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066f60 con 0x7f8338071d10 2021-03-09T10:49:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d7f0 con 0x7f8324006600 2021-03-09T10:49:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:49:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:49:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:13.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:13.392 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017080 con 0x7f8338071d10 2021-03-09T10:49:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.388+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11168 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834406c750 con 0x7f8338071d10 2021-03-09T10:49:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834406c750 con 0x7f8338071d10 2021-03-09T10:49:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:13.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405bd70 con 0x7f8338071d10 2021-03-09T10:49:13.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:49:13 smithi146 bash[13136]: debug 2021-03-09T10:49:13.391+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32619 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:49:18.394 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:49:18.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:18.395 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11168 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344014000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:18.396 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11168 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344014000).stop 2021-03-09T10:49:18.397 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:18.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:18.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:18.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:18.398 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338016110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:18.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ff70 con 0x7f8324006e60 2021-03-09T10:49:18.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f83380292c0 2021-03-09T10:49:18.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006600 2021-03-09T10:49:18.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:49:18.399 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:18.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:49:18.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:18.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338016110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:18.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:18.400 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:18.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:18.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:18.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.392+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800fd80 con 0x7f83380292c0 2021-03-09T10:49:18.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.396+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338016110 secure :-1 s=READY pgs=11177 cs=0 l=1 rev1=1 rx=0x7f833c040cb0 tx=0x7f833c079930).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:18.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa080 con 0x7f83380292c0 2021-03-09T10:49:18.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.400+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f83380292c0 2021-03-09T10:49:18.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:18.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05cd70 con 0x7f83380292c0 2021-03-09T10:49:18.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:49:18 smithi146 bash[13136]: debug 2021-03-09T10:49:18.395+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32645 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:49:23.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.396+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338016110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:49:23.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.396+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338016110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:23.401 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.396+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338016110 secure :-1 s=READY pgs=11177 cs=0 l=1 rev1=1 rx=0x7f833c040cb0 tx=0x7f833c079930).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:23.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.396+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338016110 secure :-1 s=READY pgs=11177 cs=0 l=1 rev1=1 rx=0x7f833c040cb0 tx=0x7f833c079930).stop 2021-03-09T10:49:23.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338016110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:23.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338016110 unknown :-1 s=CLOSED pgs=11177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:23.402 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:23.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:23.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.396+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338016d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:23.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d7f0 con 0x7f8324006e60 2021-03-09T10:49:23.403 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026930 con 0x7f8338071d10 2021-03-09T10:49:23.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.396+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e8f0 con 0x7f8324006600 2021-03-09T10:49:23.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.396+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:49:23.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.396+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:23.404 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.396+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:49:23.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.396+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:23.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338016d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:23.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:23.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:23.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:23.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:23.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.400+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338066f60 con 0x7f8338071d10 2021-03-09T10:49:23.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.400+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338016d10 secure :-1 s=READY pgs=11188 cs=0 l=1 rev1=1 rx=0x7f833802e100 tx=0x7f833c079930).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:23.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338071d10 2021-03-09T10:49:23.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0429c0 con 0x7f8338071d10 2021-03-09T10:49:23.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:23.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c042c50 con 0x7f8338071d10 2021-03-09T10:49:23.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:49:23 smithi146 bash[13136]: debug 2021-03-09T10:49:23.403+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32672 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:49:28.405 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338016d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:49:28.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338016d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:28.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338016d10 secure :-1 s=READY pgs=11188 cs=0 l=1 rev1=1 rx=0x7f833802e100 tx=0x7f833c079930).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:28.406 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338016d10 secure :-1 s=READY pgs=11188 cs=0 l=1 rev1=1 rx=0x7f833802e100 tx=0x7f833c079930).stop 2021-03-09T10:49:28.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f8338016d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:28.407 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f8338016d10 unknown :-1 s=CLOSED pgs=11188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:28.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833801d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:28.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:28.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:28.408 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022270 con 0x7f8324006e60 2021-03-09T10:49:28.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066150 con 0x7f83380292c0 2021-03-09T10:49:28.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ff70 con 0x7f8324006600 2021-03-09T10:49:28.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:49:28.409 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:28.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:49:28.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833801d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:28.410 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:28.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:28.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:28.411 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:28.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:28.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.404+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338074750 con 0x7f83380292c0 2021-03-09T10:49:28.412 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833801d1e0 secure :-1 s=READY pgs=11194 cs=0 l=1 rev1=1 rx=0x7f833c079900 tx=0x7f833c07c040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:28.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f83380292c0 2021-03-09T10:49:28.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.412+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c02ed90 con 0x7f83380292c0 2021-03-09T10:49:28.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:28.412+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0a2a20 con 0x7f83380292c0 2021-03-09T10:49:28.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:49:28 smithi146 bash[13136]: debug 2021-03-09T10:49:28.411+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32697 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:49:33.413 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833801d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:49:33.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833801d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:33.414 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833801d1e0 secure :-1 s=READY pgs=11194 cs=0 l=1 rev1=1 rx=0x7f833c079900 tx=0x7f833c07c040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:33.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833801d1e0 secure :-1 s=READY pgs=11194 cs=0 l=1 rev1=1 rx=0x7f833c079900 tx=0x7f833c07c040).stop 2021-03-09T10:49:33.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833801d1e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:33.415 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.408+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833801d1e0 unknown :-1 s=CLOSED pgs=11194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:33.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.408+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:33.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.408+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:33.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.408+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:33.416 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:49:33.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:33.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.408+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:49:33.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.408+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:33.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006e60 2021-03-09T10:49:33.417 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072b80 con 0x7f8338071d10 2021-03-09T10:49:33.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.408+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806bdd0 con 0x7f8324006600 2021-03-09T10:49:33.418 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.412+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:33.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.412+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:33.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.412+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:33.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.412+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:33.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.412+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:33.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.412+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338074930 con 0x7f8338071d10 2021-03-09T10:49:33.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.412+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11203 cs=0 l=1 rev1=1 rx=0x7f8344014de0 tx=0x7f8344014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:33.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.416+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344014b10 con 0x7f8338071d10 2021-03-09T10:49:33.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.416+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344014b10 con 0x7f8338071d10 2021-03-09T10:49:33.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:33.420+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344043d90 con 0x7f8338071d10 2021-03-09T10:49:33.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:49:33 smithi146 bash[13136]: debug 2021-03-09T10:49:33.415+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32725 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:49:38.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:49:38.419 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:38.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11203 cs=0 l=1 rev1=1 rx=0x7f8344014de0 tx=0x7f8344014000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:38.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11203 cs=0 l=1 rev1=1 rx=0x7f8344014de0 tx=0x7f8344014000).stop 2021-03-09T10:49:38.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:38.420 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:38.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:38.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338063080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:38.421 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:38.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005060 con 0x7f8324006e60 2021-03-09T10:49:38.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020c20 con 0x7f83380292c0 2021-03-09T10:49:38.422 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006600 2021-03-09T10:49:38.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:49:38.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:38.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:49:38.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:38.423 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338063080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:38.424 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:38.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:38.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:38.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:38.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f83380292c0 2021-03-09T10:49:38.425 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338063080 secure :-1 s=READY pgs=11214 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:38.426 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.420+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405ee90 con 0x7f83380292c0 2021-03-09T10:49:38.429 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.428+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405ee90 con 0x7f83380292c0 2021-03-09T10:49:38.430 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:38.428+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344040470 con 0x7f83380292c0 2021-03-09T10:49:38.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:49:38 smithi146 bash[13136]: debug 2021-03-09T10:49:38.422+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32750 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:49:43.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.420+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:49:43.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.420+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338063080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:43.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.420+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338063080 secure :-1 s=READY pgs=11214 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344014000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:43.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.420+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338063080 secure :-1 s=READY pgs=11214 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344014000).stop 2021-03-09T10:49:43.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.420+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338063080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:43.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.420+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338063080 unknown :-1 s=CLOSED pgs=11214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:43.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.420+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:43.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.420+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:43.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.420+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:43.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.420+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066150 con 0x7f8324006e60 2021-03-09T10:49:43.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.420+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8338071d10 2021-03-09T10:49:43.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.420+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e580 con 0x7f8324006600 2021-03-09T10:49:43.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.420+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:49:43.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.420+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:43.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.420+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:49:43.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.420+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:43.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.424+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:43.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.424+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:43.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.424+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:43.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.424+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:43.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.424+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:43.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.424+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800dfe0 con 0x7f8338071d10 2021-03-09T10:49:43.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.424+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11220 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834405f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:43.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.432+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405bd70 con 0x7f8338071d10 2021-03-09T10:49:43.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.432+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344040470 con 0x7f8338071d10 2021-03-09T10:49:43.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:43.432+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440171c0 con 0x7f8338071d10 2021-03-09T10:49:43.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:49:43 smithi146 bash[13136]: debug 2021-03-09T10:49:43.426+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32777 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:49:48.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:49:48.431 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:48.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11220 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834405f000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:48.432 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11220 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f834405f000).stop 2021-03-09T10:49:48.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:48.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:48.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:48.433 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:48.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:48.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f8324006e60 2021-03-09T10:49:48.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380180f0 con 0x7f83380292c0 2021-03-09T10:49:48.434 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005060 con 0x7f8324006600 2021-03-09T10:49:48.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:49:48.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:48.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:49:48.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:48.435 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:48.436 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:48.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:48.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:48.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:48.437 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380729a0 con 0x7f83380292c0 2021-03-09T10:49:48.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.432+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 secure :-1 s=READY pgs=11229 cs=0 l=1 rev1=1 rx=0x7f834400dfb0 tx=0x7f834405f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:48.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.436+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402b7e0 con 0x7f83380292c0 2021-03-09T10:49:48.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.436+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402b7e0 con 0x7f83380292c0 2021-03-09T10:49:48.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:48.440+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402b220 con 0x7f83380292c0 2021-03-09T10:49:48.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:49:48 smithi146 bash[13136]: debug 2021-03-09T10:49:48.434+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32803 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:49:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338077130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:49:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338077130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 secure :-1 s=READY pgs=11229 cs=0 l=1 rev1=1 rx=0x7f834400dfb0 tx=0x7f834405f000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 secure :-1 s=READY pgs=11229 cs=0 l=1 rev1=1 rx=0x7f834400dfb0 tx=0x7f834405f000).stop 2021-03-09T10:49:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338077130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 unknown :-1 s=CLOSED pgs=11229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072b80 con 0x7f8324006e60 2021-03-09T10:49:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:49:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:49:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066150 con 0x7f8338071d10 2021-03-09T10:49:53.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800bec0 con 0x7f8324006600 2021-03-09T10:49:53.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:53.443 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:53.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:53.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:53.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f8338071d10 2021-03-09T10:49:53.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11238 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c07c040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:53.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.439+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c02ed90 con 0x7f8338071d10 2021-03-09T10:49:53.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.443+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f8338071d10 2021-03-09T10:49:53.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:53.443+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338071d10 2021-03-09T10:49:53.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:49:53 smithi146 bash[13136]: debug 2021-03-09T10:49:53.442+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32830 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:49:58.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.443+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:49:58.444 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.443+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:58.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.443+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11238 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c07c040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:58.445 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.443+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11238 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c07c040).stop 2021-03-09T10:49:58.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.443+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:58.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.443+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:58.446 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.443+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:58.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.443+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:58.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.447+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:58.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.447+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f8324006e60 2021-03-09T10:49:58.447 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.447+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068290 con 0x7f83380292c0 2021-03-09T10:49:58.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.447+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380093b0 con 0x7f8324006600 2021-03-09T10:49:58.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.447+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:49:58.448 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.447+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:58.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.447+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:58.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.447+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:49:58.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.447+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:58.449 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.447+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:58.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.447+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:58.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.447+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:58.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.447+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:58.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.447+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020500 con 0x7f83380292c0 2021-03-09T10:49:58.450 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.447+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 secure :-1 s=READY pgs=11246 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c041f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:58.451 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f83380292c0 2021-03-09T10:49:58.452 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c00c8c0 con 0x7f83380292c0 2021-03-09T10:49:58.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:49:58.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0552e0 con 0x7f83380292c0 2021-03-09T10:49:58.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:49:58 smithi146 bash[13136]: debug 2021-03-09T10:49:58.446+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32856 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:50:03.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338077130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:50:03.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338077130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:03.453 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 secure :-1 s=READY pgs=11246 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c041f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:03.454 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 secure :-1 s=READY pgs=11246 cs=0 l=1 rev1=1 rx=0x7f833c040e80 tx=0x7f833c041f50).stop 2021-03-09T10:50:03.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338077130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:03.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 unknown :-1 s=CLOSED pgs=11246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:03.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:03.455 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:03.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:03.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006e60 2021-03-09T10:50:03.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005060 con 0x7f8338071d10 2021-03-09T10:50:03.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006600 2021-03-09T10:50:03.456 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:50:03.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:03.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:50:03.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:03.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.451+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:03.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:03.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:03.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:03.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:03.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802c9a0 con 0x7f8338071d10 2021-03-09T10:50:03.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11255 cs=0 l=1 rev1=1 rx=0x7f833c01cd40 tx=0x7f833c041f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:03.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338071d10 2021-03-09T10:50:03.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.459+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c770 con 0x7f8338071d10 2021-03-09T10:50:03.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:03.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0218d0 con 0x7f8338071d10 2021-03-09T10:50:03.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:50:03 smithi146 bash[13136]: debug 2021-03-09T10:50:03.454+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32883 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:50:08.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:50:08.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:08.457 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11255 cs=0 l=1 rev1=1 rx=0x7f833c01cd40 tx=0x7f833c041f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:08.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11255 cs=0 l=1 rev1=1 rx=0x7f833c01cd40 tx=0x7f833c041f50).stop 2021-03-09T10:50:08.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:08.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:08.458 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:08.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380201c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:08.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.455+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:08.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066150 con 0x7f8324006e60 2021-03-09T10:50:08.459 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380172d0 con 0x7f83380292c0 2021-03-09T10:50:08.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.455+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380180f0 con 0x7f8324006600 2021-03-09T10:50:08.460 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.455+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:50:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.455+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.459+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380201c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.459+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:50:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.459+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.459+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.459+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.459+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.459+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.459+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806bd80 con 0x7f83380292c0 2021-03-09T10:50:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.459+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380201c0 secure :-1 s=READY pgs=11264 cs=0 l=1 rev1=1 rx=0x7f833801b200 tx=0x7f834405f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f83380292c0 2021-03-09T10:50:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f83380292c0 2021-03-09T10:50:08.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:08.467+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402b220 con 0x7f83380292c0 2021-03-09T10:50:08.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:50:08 smithi146 bash[13136]: debug 2021-03-09T10:50:08.458+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32909 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:50:13.462 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380201c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:50:13.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380201c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:13.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380201c0 secure :-1 s=READY pgs=11264 cs=0 l=1 rev1=1 rx=0x7f833801b200 tx=0x7f834405f000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:13.463 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380201c0 secure :-1 s=READY pgs=11264 cs=0 l=1 rev1=1 rx=0x7f833801b200 tx=0x7f834405f000).stop 2021-03-09T10:50:13.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380201c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:13.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380201c0 unknown :-1 s=CLOSED pgs=11264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:13.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:13.464 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:13.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:13.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017d70 con 0x7f8324006e60 2021-03-09T10:50:13.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005060 con 0x7f8338071d10 2021-03-09T10:50:13.465 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006600 2021-03-09T10:50:13.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:50:13.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:13.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:50:13.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:13.466 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:13.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:13.467 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:13.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:13.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:13.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.463+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380170d0 con 0x7f8338071d10 2021-03-09T10:50:13.468 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.467+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11272 cs=0 l=1 rev1=1 rx=0x7f833c040cb0 tx=0x7f833c041f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:13.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.471+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa080 con 0x7f8338071d10 2021-03-09T10:50:13.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.475+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079820 con 0x7f8338071d10 2021-03-09T10:50:13.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:13.475+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01cdb0 con 0x7f8338071d10 2021-03-09T10:50:13.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:50:13 smithi146 bash[13136]: debug 2021-03-09T10:50:13.466+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32936 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:50:18.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.467+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:50:18.469 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.467+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:18.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.467+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11272 cs=0 l=1 rev1=1 rx=0x7f833c040cb0 tx=0x7f833c041f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:18.470 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.467+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11272 cs=0 l=1 rev1=1 rx=0x7f833c040cb0 tx=0x7f833c041f50).stop 2021-03-09T10:50:18.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.467+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:18.471 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.467+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:18.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.471+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:18.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.471+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:18.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.471+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:18.472 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.471+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066150 con 0x7f8324006e60 2021-03-09T10:50:18.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.471+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f83380292c0 2021-03-09T10:50:18.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.471+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d930 con 0x7f8324006600 2021-03-09T10:50:18.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.471+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:50:18.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.471+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:18.473 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.471+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:50:18.474 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.471+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:18.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.471+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:18.475 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.475+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:18.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.475+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:18.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.475+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:18.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.475+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:18.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.475+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800dfe0 con 0x7f83380292c0 2021-03-09T10:50:18.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.475+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 secure :-1 s=READY pgs=11284 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c041f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:18.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.479+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f83380292c0 2021-03-09T10:50:18.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.479+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c062890 con 0x7f83380292c0 2021-03-09T10:50:18.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:18.479+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0648e0 con 0x7f83380292c0 2021-03-09T10:50:18.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:50:18 smithi146 bash[13136]: debug 2021-03-09T10:50:18.470+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32962 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:50:23.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.475+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338077130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:50:23.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.475+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338077130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:23.476 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.475+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 secure :-1 s=READY pgs=11284 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c041f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:23.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.475+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 secure :-1 s=READY pgs=11284 cs=0 l=1 rev1=1 rx=0x7f833c05e280 tx=0x7f833c041f50).stop 2021-03-09T10:50:23.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.475+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338077130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:23.477 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.475+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 unknown :-1 s=CLOSED pgs=11284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:23.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.475+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:23.478 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.475+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:23.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.475+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:23.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.475+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022270 con 0x7f8324006e60 2021-03-09T10:50:23.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.475+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8338071d10 2021-03-09T10:50:23.479 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.475+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dac0 con 0x7f8324006600 2021-03-09T10:50:23.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.475+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:50:23.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.475+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:23.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.475+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:50:23.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.479+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:23.480 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.479+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:23.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.479+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:23.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.479+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:23.481 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.479+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:23.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.479+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:23.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.479+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020860 con 0x7f8338071d10 2021-03-09T10:50:23.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.479+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11290 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:23.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.483+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338071d10 2021-03-09T10:50:23.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.483+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c026750 con 0x7f8338071d10 2021-03-09T10:50:23.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:23.487+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c057cd0 con 0x7f8338071d10 2021-03-09T10:50:23.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:50:23 smithi146 bash[13136]: debug 2021-03-09T10:50:23.478+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 32989 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:50:28.482 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:50:28.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:28.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11290 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:28.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11290 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c02ed20).stop 2021-03-09T10:50:28.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:28.483 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:28.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:28.484 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:28.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:28.485 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005060 con 0x7f8324006e60 2021-03-09T10:50:28.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f83380292c0 2021-03-09T10:50:28.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066150 con 0x7f8324006600 2021-03-09T10:50:28.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:50:28.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:28.486 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:50:28.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:28.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:28.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:28.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:28.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:28.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:28.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801ccc0 con 0x7f83380292c0 2021-03-09T10:50:28.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.483+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 secure :-1 s=READY pgs=11298 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:28.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.487+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f83380292c0 2021-03-09T10:50:28.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.491+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c064540 con 0x7f83380292c0 2021-03-09T10:50:28.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:28.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f83380292c0 2021-03-09T10:50:28.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:50:28 smithi146 bash[13136]: debug 2021-03-09T10:50:28.482+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33015 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:50:33.487 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338077130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:50:33.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338077130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:33.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 secure :-1 s=READY pgs=11298 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:33.488 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 secure :-1 s=READY pgs=11298 cs=0 l=1 rev1=1 rx=0x7f833c04e8e0 tx=0x7f833c02ed20).stop 2021-03-09T10:50:33.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338077130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:33.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338077130 unknown :-1 s=CLOSED pgs=11298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:33.489 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:33.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:33.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:33.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017d70 con 0x7f8324006e60 2021-03-09T10:50:33.490 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8338071d10 2021-03-09T10:50:33.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dac0 con 0x7f8324006600 2021-03-09T10:50:33.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:50:33.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:33.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:50:33.491 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:33.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.487+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:33.492 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.491+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:33.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.491+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:33.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.491+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:33.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.491+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:33.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.491+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380748f0 con 0x7f8338071d10 2021-03-09T10:50:33.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.491+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11307 cs=0 l=1 rev1=1 rx=0x7f833c01aa80 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:33.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05c0a0 con 0x7f8338071d10 2021-03-09T10:50:33.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01d710 con 0x7f8338071d10 2021-03-09T10:50:33.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:33.499+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f8338071d10 2021-03-09T10:50:33.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:50:33 smithi146 bash[13136]: debug 2021-03-09T10:50:33.490+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33042 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:50:38.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.491+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:50:38.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.491+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:38.493 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.491+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11307 cs=0 l=1 rev1=1 rx=0x7f833c01aa80 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:38.494 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.491+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11307 cs=0 l=1 rev1=1 rx=0x7f833c01aa80 tx=0x7f833c02ed20).stop 2021-03-09T10:50:38.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:38.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:38.495 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:38.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:38.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:38.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060bc0 con 0x7f8324006e60 2021-03-09T10:50:38.496 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f83380292c0 2021-03-09T10:50:38.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338066150 con 0x7f8324006600 2021-03-09T10:50:38.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:50:38.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:38.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:50:38.497 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:38.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:38.498 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:38.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:38.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:38.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:38.499 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f83380292c0 2021-03-09T10:50:38.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.495+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800ce90 secure :-1 s=READY pgs=11316 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834405f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:38.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404aa60 con 0x7f83380292c0 2021-03-09T10:50:38.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.503+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f83380292c0 2021-03-09T10:50:38.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:38.507+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405bd70 con 0x7f83380292c0 2021-03-09T10:50:38.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:50:38 smithi146 bash[13136]: debug 2021-03-09T10:50:38.494+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33068 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:50:43.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:50:43.500 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:43.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800ce90 secure :-1 s=READY pgs=11316 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834405f000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:43.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800ce90 secure :-1 s=READY pgs=11316 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f834405f000).stop 2021-03-09T10:50:43.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800ce90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:43.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800ce90 unknown :-1 s=CLOSED pgs=11316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:43.501 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:43.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:43.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:43.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d930 con 0x7f8324006e60 2021-03-09T10:50:43.502 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022270 con 0x7f8338071d10 2021-03-09T10:50:43.503 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324006600 2021-03-09T10:50:43.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:50:43.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:43.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:50:43.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:43.504 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.498+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:43.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.502+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:43.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.502+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:43.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.502+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:43.505 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.502+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:43.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.502+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f8338071d10 2021-03-09T10:50:43.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.502+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11324 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f834405f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:43.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.506+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405d1d0 con 0x7f8338071d10 2021-03-09T10:50:43.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.506+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405d1d0 con 0x7f8338071d10 2021-03-09T10:50:43.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:43.506+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344079ba0 con 0x7f8338071d10 2021-03-09T10:50:43.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:50:43 smithi146 bash[13136]: debug 2021-03-09T10:50:43.502+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33094 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:50:48.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:50:48.506 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:48.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11324 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f834405f000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:48.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 secure :-1 s=READY pgs=11324 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f834405f000).stop 2021-03-09T10:50:48.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:48.507 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338071d10 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=11324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:48.508 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:48.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:48.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:48.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063d90 con 0x7f8324006e60 2021-03-09T10:50:48.509 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dac0 con 0x7f83380292c0 2021-03-09T10:50:48.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006600 2021-03-09T10:50:48.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:50:48.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:48.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:48.510 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:50:48.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:48.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:48.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:48.511 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:48.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:48.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f83380292c0 2021-03-09T10:50:48.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.506+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800f2b0 secure :-1 s=READY pgs=11333 cs=0 l=1 rev1=1 rx=0x7f833c0a8b60 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:48.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c04a300 con 0x7f83380292c0 2021-03-09T10:50:48.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0252e0 con 0x7f83380292c0 2021-03-09T10:50:48.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:48.514+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02ed90 con 0x7f83380292c0 2021-03-09T10:50:49.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:50:48 smithi146 bash[13136]: debug 2021-03-09T10:50:48.505+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33121 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:50:53.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.510+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:50:53.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.510+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:53.512 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.510+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800f2b0 secure :-1 s=READY pgs=11333 cs=0 l=1 rev1=1 rx=0x7f833c0a8b60 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:53.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.510+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800f2b0 secure :-1 s=READY pgs=11333 cs=0 l=1 rev1=1 rx=0x7f833c0a8b60 tx=0x7f833c02ed20).stop 2021-03-09T10:50:53.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800f2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:53.513 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800f2b0 unknown :-1 s=CLOSED pgs=11333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:53.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:53.514 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:53.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.510+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:53.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338005060 con 0x7f8324006e60 2021-03-09T10:50:53.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380095f0 con 0x7f833805e5b0 2021-03-09T10:50:53.515 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.510+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010830 con 0x7f8324006600 2021-03-09T10:50:53.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.510+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:50:53.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.510+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:53.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.510+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:50:53.516 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.510+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:53.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.514+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:53.517 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.514+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:53.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.514+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:53.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.514+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:53.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.514+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:53.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.514+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802b6b0 con 0x7f833805e5b0 2021-03-09T10:50:53.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.514+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071920 secure :-1 s=READY pgs=11342 cs=0 l=1 rev1=1 rx=0x7f8338034b90 tx=0x7f834405f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:53.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405d480 con 0x7f833805e5b0 2021-03-09T10:50:53.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833805e5b0 2021-03-09T10:50:53.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:53.522+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833805e5b0 2021-03-09T10:50:54.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:50:53 smithi146 bash[13136]: debug 2021-03-09T10:50:53.513+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33147 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:50:58.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:50:58.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:58.518 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071920 secure :-1 s=READY pgs=11342 cs=0 l=1 rev1=1 rx=0x7f8338034b90 tx=0x7f834405f000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:58.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071920 secure :-1 s=READY pgs=11342 cs=0 l=1 rev1=1 rx=0x7f8338034b90 tx=0x7f834405f000).stop 2021-03-09T10:50:58.519 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:58.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071920 unknown :-1 s=CLOSED pgs=11342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:58.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:58.520 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338029e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:58.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:58.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fea0 con 0x7f8324006e60 2021-03-09T10:50:58.521 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380093b0 con 0x7f83380292c0 2021-03-09T10:50:58.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006600 2021-03-09T10:50:58.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:50:58.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:58.522 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:50:58.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:58.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.518+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338029e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:58.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.522+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:58.523 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.522+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:58.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.522+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:58.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.522+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:58.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.522+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802e8a0 con 0x7f83380292c0 2021-03-09T10:50:58.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.522+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338029e10 secure :-1 s=READY pgs=11351 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f834405f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:58.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.526+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404af30 con 0x7f83380292c0 2021-03-09T10:50:58.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.526+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404af30 con 0x7f83380292c0 2021-03-09T10:50:58.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:50:58.530+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402c2c0 con 0x7f83380292c0 2021-03-09T10:50:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:50:58 smithi146 bash[13136]: debug 2021-03-09T10:50:58.521+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33174 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:51:03.524 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.522+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338029e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:51:03.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.522+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338029e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:03.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.522+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338029e10 secure :-1 s=READY pgs=11351 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f834405f000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:03.525 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.522+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338029e10 secure :-1 s=READY pgs=11351 cs=0 l=1 rev1=1 rx=0x7f834404c370 tx=0x7f834405f000).stop 2021-03-09T10:51:03.526 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.522+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338029e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:03.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.522+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338029e10 unknown :-1 s=CLOSED pgs=11351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:03.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.522+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:03.527 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.522+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:03.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.522+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:03.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.522+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f8324006e60 2021-03-09T10:51:03.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.522+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060bc0 con 0x7f833805e5b0 2021-03-09T10:51:03.528 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.522+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006600 2021-03-09T10:51:03.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.522+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:51:03.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.522+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:03.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.522+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:51:03.529 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.522+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:03.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.526+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:03.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.526+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:03.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.526+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:03.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.526+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:03.530 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.526+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:03.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.526+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338011780 con 0x7f833805e5b0 2021-03-09T10:51:03.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.526+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11359 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834405f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:03.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.530+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405dba0 con 0x7f833805e5b0 2021-03-09T10:51:03.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.530+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405d9c0 con 0x7f833805e5b0 2021-03-09T10:51:03.534 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:03.534+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405bd70 con 0x7f833805e5b0 2021-03-09T10:51:04.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:51:03 smithi146 bash[13136]: debug 2021-03-09T10:51:03.525+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33200 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:51:08.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:51:08.531 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:08.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11359 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834405f000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:08.532 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11359 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f834405f000).stop 2021-03-09T10:51:08.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:08.533 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=CLOSED pgs=11359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:08.535 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:08.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:08.536 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:08.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324006e60 2021-03-09T10:51:08.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f83380292c0 2021-03-09T10:51:08.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006600 2021-03-09T10:51:08.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:51:08.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:08.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:51:08.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:08.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:08.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:08.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:08.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:08.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:08.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.530+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338061ca0 con 0x7f83380292c0 2021-03-09T10:51:08.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.534+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800f2b0 secure :-1 s=READY pgs=11368 cs=0 l=1 rev1=1 rx=0x7f833c09a350 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:08.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.534+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f83380292c0 2021-03-09T10:51:08.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.538+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079630 con 0x7f83380292c0 2021-03-09T10:51:08.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:08.538+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01ad20 con 0x7f83380292c0 2021-03-09T10:51:09.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:51:08 smithi146 bash[13136]: debug 2021-03-09T10:51:08.533+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33227 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:51:13.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.534+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:51:13.537 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.534+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:13.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.534+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800f2b0 secure :-1 s=READY pgs=11368 cs=0 l=1 rev1=1 rx=0x7f833c09a350 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:13.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.534+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800f2b0 secure :-1 s=READY pgs=11368 cs=0 l=1 rev1=1 rx=0x7f833c09a350 tx=0x7f833c02ed20).stop 2021-03-09T10:51:13.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.534+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800f2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:13.538 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.534+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800f2b0 unknown :-1 s=CLOSED pgs=11368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:13.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.534+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:13.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.534+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:13.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.534+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338066b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:13.539 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.534+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380123f0 con 0x7f8324006e60 2021-03-09T10:51:13.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.534+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f833805e5b0 2021-03-09T10:51:13.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.534+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060bc0 con 0x7f8324006600 2021-03-09T10:51:13.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.534+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:51:13.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.534+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:51:13.540 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.534+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:13.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.534+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:13.541 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.538+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338066b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:13.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.538+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:13.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.538+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:13.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.538+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:13.542 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.538+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:13.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.538+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028fa0 con 0x7f833805e5b0 2021-03-09T10:51:13.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.538+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338066b60 secure :-1 s=READY pgs=11376 cs=0 l=1 rev1=1 rx=0x7f833c0795d0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:13.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.542+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c036340 con 0x7f833805e5b0 2021-03-09T10:51:13.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.542+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079630 con 0x7f833805e5b0 2021-03-09T10:51:13.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:13.546+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f833805e5b0 2021-03-09T10:51:14.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:51:13 smithi146 bash[13136]: debug 2021-03-09T10:51:13.537+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33252 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:51:18.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338066b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:51:18.543 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338066b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:18.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338066b60 secure :-1 s=READY pgs=11376 cs=0 l=1 rev1=1 rx=0x7f833c0795d0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:18.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338066b60 secure :-1 s=READY pgs=11376 cs=0 l=1 rev1=1 rx=0x7f833c0795d0 tx=0x7f833c02ed20).stop 2021-03-09T10:51:18.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338066b60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:18.544 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338066b60 unknown :-1 s=CLOSED pgs=11376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:18.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:18.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:18.545 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:18.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f8324006e60 2021-03-09T10:51:18.546 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f83380292c0 2021-03-09T10:51:18.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338019f30 con 0x7f8324006600 2021-03-09T10:51:18.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:51:18.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:18.547 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:51:18.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:18.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.542+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:18.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.546+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:18.548 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.546+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:18.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.546+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:18.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.546+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:18.549 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.546+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ae20 con 0x7f83380292c0 2021-03-09T10:51:18.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.546+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11385 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834405f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:18.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.550+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f83380292c0 2021-03-09T10:51:18.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.550+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440105c0 con 0x7f83380292c0 2021-03-09T10:51:18.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:18.554+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f83380292c0 2021-03-09T10:51:19.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:51:18 smithi146 bash[13136]: debug 2021-03-09T10:51:18.545+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33280 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:51:23.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.546+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:51:23.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.546+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:23.550 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.546+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11385 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834405f000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:23.551 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.550+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11385 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f834405f000).stop 2021-03-09T10:51:23.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.550+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:23.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.550+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=CLOSED pgs=11385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:23.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.550+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:23.552 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.550+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:23.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.550+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:23.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.550+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c9a0 con 0x7f8324006e60 2021-03-09T10:51:23.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.550+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022270 con 0x7f833805e5b0 2021-03-09T10:51:23.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.550+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324006600 2021-03-09T10:51:23.553 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.550+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:51:23.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.550+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:23.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.550+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:51:23.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.550+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:23.554 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.550+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:23.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.554+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:23.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.554+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:23.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.554+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:23.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.554+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:23.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.554+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026af0 con 0x7f833805e5b0 2021-03-09T10:51:23.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.554+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11396 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834405f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:23.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.554+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440112f0 con 0x7f833805e5b0 2021-03-09T10:51:23.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.558+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440112f0 con 0x7f833805e5b0 2021-03-09T10:51:23.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:23.558+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440171c0 con 0x7f833805e5b0 2021-03-09T10:51:24.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:51:23 smithi146 bash[13136]: debug 2021-03-09T10:51:23.549+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33305 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:51:28.555 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.553+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:51:28.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.553+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:28.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.553+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11396 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834405f000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:28.556 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.553+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11396 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f834405f000).stop 2021-03-09T10:51:28.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.553+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:28.557 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.553+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=CLOSED pgs=11396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:28.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.553+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:28.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.553+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:28.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.553+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:28.558 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.553+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006e60 2021-03-09T10:51:28.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.553+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f83380292c0 2021-03-09T10:51:28.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.553+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060bc0 con 0x7f8324006600 2021-03-09T10:51:28.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.553+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:51:28.559 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.553+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:28.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.553+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:51:28.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.553+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:28.560 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.557+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:28.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.557+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:28.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.557+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:28.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.557+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:28.561 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.557+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:28.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.557+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800ae20 con 0x7f83380292c0 2021-03-09T10:51:28.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.557+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11404 cs=0 l=1 rev1=1 rx=0x7f834403d040 tx=0x7f8344027040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:28.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.561+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440112f0 con 0x7f83380292c0 2021-03-09T10:51:28.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.561+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440112f0 con 0x7f83380292c0 2021-03-09T10:51:28.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:28.565+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344010330 con 0x7f83380292c0 2021-03-09T10:51:29.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:51:28 smithi146 bash[13136]: debug 2021-03-09T10:51:28.557+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33333 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:51:33.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:51:33.562 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:33.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11404 cs=0 l=1 rev1=1 rx=0x7f834403d040 tx=0x7f8344027040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:33.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11404 cs=0 l=1 rev1=1 rx=0x7f834403d040 tx=0x7f8344027040).stop 2021-03-09T10:51:33.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:33.563 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=CLOSED pgs=11404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:33.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:33.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:33.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:33.564 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f8324006e60 2021-03-09T10:51:33.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380060b0 con 0x7f833805e5b0 2021-03-09T10:51:33.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023340 con 0x7f8324006600 2021-03-09T10:51:33.565 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:51:33.566 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:33.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:33.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:51:33.567 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:33.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:33.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:33.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:33.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:33.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.561+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380748f0 con 0x7f833805e5b0 2021-03-09T10:51:33.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.565+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11411 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:33.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.569+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f833805e5b0 2021-03-09T10:51:33.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.569+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079630 con 0x7f833805e5b0 2021-03-09T10:51:33.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:33.569+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f833805e5b0 2021-03-09T10:51:34.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:51:33 smithi146 bash[13136]: debug 2021-03-09T10:51:33.561+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33358 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:51:38.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:51:38.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:38.568 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11411 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:38.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11411 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c02ed20).stop 2021-03-09T10:51:38.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:38.569 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=CLOSED pgs=11411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:38.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:38.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:38.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:38.570 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f8324006e60 2021-03-09T10:51:38.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060bc0 con 0x7f83380292c0 2021-03-09T10:51:38.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064d40 con 0x7f8324006600 2021-03-09T10:51:38.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:51:38.571 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:38.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:51:38.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:38.572 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.565+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:38.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.569+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:38.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.569+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:38.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.569+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:38.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.569+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:38.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.569+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380061f0 con 0x7f83380292c0 2021-03-09T10:51:38.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.569+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11420 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344027040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:38.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.573+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344054d80 con 0x7f83380292c0 2021-03-09T10:51:38.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.573+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344054d80 con 0x7f83380292c0 2021-03-09T10:51:38.577 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:38.573+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405bd70 con 0x7f83380292c0 2021-03-09T10:51:39.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:51:38 smithi146 bash[13136]: debug 2021-03-09T10:51:38.569+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33386 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:51:43.573 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:51:43.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:43.574 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11420 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344027040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:43.575 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11420 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344027040).stop 2021-03-09T10:51:43.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:43.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=CLOSED pgs=11420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:43.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:43.579 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:43.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:43.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006e60 2021-03-09T10:51:43.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f833805e5b0 2021-03-09T10:51:43.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023580 con 0x7f8324006600 2021-03-09T10:51:43.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:51:43.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:51:43.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:43.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:43.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:43.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:43.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:43.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:43.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:43.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.573+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338026af0 con 0x7f833805e5b0 2021-03-09T10:51:43.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.577+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11428 cs=0 l=1 rev1=1 rx=0x7f833c02f8c0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:43.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.577+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f833805e5b0 2021-03-09T10:51:43.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.577+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c02ed90 con 0x7f833805e5b0 2021-03-09T10:51:43.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:43.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f833805e5b0 2021-03-09T10:51:44.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:51:43 smithi146 bash[13136]: debug 2021-03-09T10:51:43.573+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33411 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:51:48.580 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:51:48.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:48.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11428 cs=0 l=1 rev1=1 rx=0x7f833c02f8c0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:48.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11428 cs=0 l=1 rev1=1 rx=0x7f833c02f8c0 tx=0x7f833c02ed20).stop 2021-03-09T10:51:48.581 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:48.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=CLOSED pgs=11428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:48.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:48.582 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:48.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:48.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dfd0 con 0x7f8324006e60 2021-03-09T10:51:48.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800c2e0 con 0x7f83380292c0 2021-03-09T10:51:48.583 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006600 2021-03-09T10:51:48.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:51:48.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:48.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:51:48.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:48.584 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:48.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:48.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:48.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:48.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:48.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017480 con 0x7f83380292c0 2021-03-09T10:51:48.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.581+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11439 cs=0 l=1 rev1=1 rx=0x7f833c09a350 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:48.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa080 con 0x7f83380292c0 2021-03-09T10:51:48.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c075d60 con 0x7f83380292c0 2021-03-09T10:51:48.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:48.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02f3b0 con 0x7f83380292c0 2021-03-09T10:51:49.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:51:48 smithi146 bash[13136]: debug 2021-03-09T10:51:48.581+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33439 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:51:53.585 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:51:53.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:53.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11439 cs=0 l=1 rev1=1 rx=0x7f833c09a350 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:53.586 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11439 cs=0 l=1 rev1=1 rx=0x7f833c09a350 tx=0x7f833c02ed20).stop 2021-03-09T10:51:53.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:53.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=CLOSED pgs=11439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:53.587 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:53.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:53.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:53.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f8324006e60 2021-03-09T10:51:53.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064d40 con 0x7f833805e5b0 2021-03-09T10:51:53.588 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006600 2021-03-09T10:51:53.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:51:53.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:53.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:51:53.589 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:53.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:53.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:53.590 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:53.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:53.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:53.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.585+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806e5a0 con 0x7f833805e5b0 2021-03-09T10:51:53.591 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11446 cs=0 l=1 rev1=1 rx=0x7f833c0798b0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:53.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c069ed0 con 0x7f833805e5b0 2021-03-09T10:51:53.595 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0414c0 con 0x7f833805e5b0 2021-03-09T10:51:53.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:53.593+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833805e5b0 2021-03-09T10:51:54.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:51:53 smithi146 bash[13136]: debug 2021-03-09T10:51:53.588+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33464 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:51:58.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.589+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:51:58.593 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.589+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:58.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11446 cs=0 l=1 rev1=1 rx=0x7f833c0798b0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:58.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.589+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11446 cs=0 l=1 rev1=1 rx=0x7f833c0798b0 tx=0x7f833c02ed20).stop 2021-03-09T10:51:58.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:58.594 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=CLOSED pgs=11446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:58.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833806dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:58.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:58.596 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.589+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:58.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073f20 con 0x7f8324006e60 2021-03-09T10:51:58.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801dfd0 con 0x7f83380292c0 2021-03-09T10:51:58.597 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.589+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338072d80 con 0x7f8324006600 2021-03-09T10:51:58.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.593+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:51:58.598 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.593+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:58.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:51:58.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:58.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833806dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:58.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:58.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:58.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:58.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:58.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.593+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380117b0 con 0x7f83380292c0 2021-03-09T10:51:58.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.593+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833806dce0 secure :-1 s=READY pgs=11454 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:58.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f83380292c0 2021-03-09T10:51:58.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0267f0 con 0x7f83380292c0 2021-03-09T10:51:58.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:51:58.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c064000 con 0x7f83380292c0 2021-03-09T10:51:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:51:58 smithi146 bash[13136]: debug 2021-03-09T10:51:58.592+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33492 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:52:03.600 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833806dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:52:03.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833806dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:03.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833806dce0 secure :-1 s=READY pgs=11454 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:03.601 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833806dce0 secure :-1 s=READY pgs=11454 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c02ed20).stop 2021-03-09T10:52:03.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833806dce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:03.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833806dce0 unknown :-1 s=CLOSED pgs=11454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:03.602 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:03.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:03.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:52:03.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:03.603 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338002a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:03.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006e60 2021-03-09T10:52:03.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f833805e5b0 2021-03-09T10:52:03.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006600 2021-03-09T10:52:03.604 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:52:03.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:03.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.597+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338002a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:03.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.601+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:03.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.601+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:03.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.601+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:03.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.601+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:03.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.601+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028fa0 con 0x7f833805e5b0 2021-03-09T10:52:03.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.601+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338002a70 secure :-1 s=READY pgs=11463 cs=0 l=1 rev1=1 rx=0x7f833c02a010 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:03.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f833805e5b0 2021-03-09T10:52:03.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0a2bc0 con 0x7f833805e5b0 2021-03-09T10:52:03.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:03.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f833805e5b0 2021-03-09T10:52:04.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:52:03 smithi146 bash[13136]: debug 2021-03-09T10:52:03.600+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33517 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:52:08.605 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.601+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338002a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:52:08.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.601+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338002a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:08.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.601+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338002a70 secure :-1 s=READY pgs=11463 cs=0 l=1 rev1=1 rx=0x7f833c02a010 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:08.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.601+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338002a70 secure :-1 s=READY pgs=11463 cs=0 l=1 rev1=1 rx=0x7f833c02a010 tx=0x7f833c02ed20).stop 2021-03-09T10:52:08.606 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.601+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338002a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:08.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.601+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338002a70 unknown :-1 s=CLOSED pgs=11463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:08.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.601+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:08.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.601+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380098e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:08.607 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.601+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:08.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800fb90 con 0x7f8324006e60 2021-03-09T10:52:08.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068ac0 con 0x7f83380292c0 2021-03-09T10:52:08.608 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.605+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380262c0 con 0x7f8324006600 2021-03-09T10:52:08.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.605+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:52:08.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.605+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:08.609 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.605+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:52:08.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.605+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:08.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.605+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:08.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.605+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:08.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.605+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:08.610 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.605+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:08.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.605+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:08.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.605+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801a7d0 con 0x7f83380292c0 2021-03-09T10:52:08.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.609+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380098e0 secure :-1 s=READY pgs=11473 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344027040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:08.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f83380292c0 2021-03-09T10:52:08.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.609+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400d450 con 0x7f83380292c0 2021-03-09T10:52:08.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:08.613+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402aa80 con 0x7f83380292c0 2021-03-09T10:52:09.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:52:08 smithi146 bash[13136]: debug 2021-03-09T10:52:08.604+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33545 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:52:13.611 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:52:13.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:13.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380098e0 secure :-1 s=READY pgs=11473 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344027040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:13.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380098e0 secure :-1 s=READY pgs=11473 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344027040).stop 2021-03-09T10:52:13.612 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380098e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380098e0 unknown :-1 s=CLOSED pgs=11473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:13.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:52:13.614 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:13.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:13.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006e60 2021-03-09T10:52:13.615 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012250 con 0x7f833805e5b0 2021-03-09T10:52:13.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068ac0 con 0x7f8324006600 2021-03-09T10:52:13.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:52:13.616 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:13.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:13.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:13.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:13.617 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:13.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:13.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338005010 con 0x7f833805e5b0 2021-03-09T10:52:13.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.612+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11480 cs=0 l=1 rev1=1 rx=0x7f8344079ca0 tx=0x7f8344027040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:13.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f833805e5b0 2021-03-09T10:52:13.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f833805e5b0 2021-03-09T10:52:13.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:13.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344017310 con 0x7f833805e5b0 2021-03-09T10:52:14.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:52:13 smithi146 bash[13136]: debug 2021-03-09T10:52:13.612+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33570 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:52:18.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:52:18.618 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:18.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11480 cs=0 l=1 rev1=1 rx=0x7f8344079ca0 tx=0x7f8344027040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:18.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11480 cs=0 l=1 rev1=1 rx=0x7f8344079ca0 tx=0x7f8344027040).stop 2021-03-09T10:52:18.619 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:18.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=CLOSED pgs=11480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:18.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:18.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:18.620 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380098e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:18.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006e60 2021-03-09T10:52:18.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f83380292c0 2021-03-09T10:52:18.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006600 2021-03-09T10:52:18.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:52:18.621 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:18.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:52:18.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:18.622 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:18.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:18.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:18.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:18.623 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:18.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028480 con 0x7f83380292c0 2021-03-09T10:52:18.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.616+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380098e0 secure :-1 s=READY pgs=11489 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:18.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f83380292c0 2021-03-09T10:52:18.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c036340 con 0x7f83380292c0 2021-03-09T10:52:18.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:18.624+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f83380292c0 2021-03-09T10:52:19.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:52:18 smithi146 bash[13136]: debug 2021-03-09T10:52:18.620+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33598 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:52:23.624 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:52:23.625 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:23.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380098e0 secure :-1 s=READY pgs=11489 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:23.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380098e0 secure :-1 s=READY pgs=11489 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c02ed20).stop 2021-03-09T10:52:23.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380098e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:23.626 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380098e0 unknown :-1 s=CLOSED pgs=11489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:23.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f833800a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:23.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:23.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:23.627 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068cb0 con 0x7f8324006e60 2021-03-09T10:52:23.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801cd10 con 0x7f833805e5b0 2021-03-09T10:52:23.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b8f0 con 0x7f8324006600 2021-03-09T10:52:23.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:52:23.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:52:23.628 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:23.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:23.629 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.620+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f833800a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:23.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.624+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:23.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.624+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:23.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.624+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:23.630 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.624+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:23.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.624+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338028480 con 0x7f833805e5b0 2021-03-09T10:52:23.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.624+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f833800a630 secure :-1 s=READY pgs=11498 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:23.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.628+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f833805e5b0 2021-03-09T10:52:23.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.628+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c010350 con 0x7f833805e5b0 2021-03-09T10:52:23.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:23.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0543f0 con 0x7f833805e5b0 2021-03-09T10:52:24.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:52:23 smithi146 bash[13136]: debug 2021-03-09T10:52:23.624+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33623 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:52:28.631 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f833800a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:52:28.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f833800a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:28.632 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f833800a630 secure :-1 s=READY pgs=11498 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:28.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f833800a630 secure :-1 s=READY pgs=11498 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c02ed20).stop 2021-03-09T10:52:28.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f833800a630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:28.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f833800a630 unknown :-1 s=CLOSED pgs=11498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:28.633 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:28.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:28.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:28.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017250 con 0x7f8324006e60 2021-03-09T10:52:28.634 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f83380292c0 2021-03-09T10:52:28.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b8f0 con 0x7f8324006600 2021-03-09T10:52:28.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:52:28.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:28.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:52:28.635 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:28.636 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:28.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:28.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:28.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:28.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:28.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f83380292c0 2021-03-09T10:52:28.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.632+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802d6a0 secure :-1 s=READY pgs=11507 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344027040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:28.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403cd20 con 0x7f83380292c0 2021-03-09T10:52:28.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.640+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f83380292c0 2021-03-09T10:52:28.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:28.644+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405bd70 con 0x7f83380292c0 2021-03-09T10:52:29.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:52:28 smithi146 bash[13136]: debug 2021-03-09T10:52:28.632+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33651 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:52:33.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833802d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:52:33.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833802d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:33.637 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802d6a0 secure :-1 s=READY pgs=11507 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344027040).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:33.638 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802d6a0 secure :-1 s=READY pgs=11507 cs=0 l=1 rev1=1 rx=0x7f834404aa10 tx=0x7f8344027040).stop 2021-03-09T10:52:33.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833802d6a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:33.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802d6a0 unknown :-1 s=CLOSED pgs=11507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:33.639 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:33.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:33.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:33.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022270 con 0x7f8324006e60 2021-03-09T10:52:33.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020db0 con 0x7f833805e5b0 2021-03-09T10:52:33.640 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006600 2021-03-09T10:52:33.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:52:33.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:33.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:52:33.641 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:33.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:33.642 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:33.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:33.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:33.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:33.643 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380191f0 con 0x7f833805e5b0 2021-03-09T10:52:33.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.636+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11516 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f83440032d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:33.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.640+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344069be0 con 0x7f833805e5b0 2021-03-09T10:52:33.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.644+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f833805e5b0 2021-03-09T10:52:33.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:33.644+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440179e0 con 0x7f833805e5b0 2021-03-09T10:52:34.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:52:33 smithi146 bash[13136]: debug 2021-03-09T10:52:33.636+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33676 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:52:38.644 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:52:38.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:38.645 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11516 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f83440032d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:38.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11516 cs=0 l=1 rev1=1 rx=0x7f8344004fd0 tx=0x7f83440032d0).stop 2021-03-09T10:52:38.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:38.646 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=CLOSED pgs=11516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:38.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:38.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338019650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:38.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:38.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020920 con 0x7f8324006e60 2021-03-09T10:52:38.647 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b070 con 0x7f83380292c0 2021-03-09T10:52:38.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023a70 con 0x7f8324006600 2021-03-09T10:52:38.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:52:38.648 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:38.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:52:38.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:38.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.640+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338019650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:38.649 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.644+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:38.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.644+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:38.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.644+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:38.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.644+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:38.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.644+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338009fc0 con 0x7f83380292c0 2021-03-09T10:52:38.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.644+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338019650 secure :-1 s=READY pgs=11524 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:38.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404b510 con 0x7f83380292c0 2021-03-09T10:52:38.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.648+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834404b510 con 0x7f83380292c0 2021-03-09T10:52:38.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:38.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f83380292c0 2021-03-09T10:52:39.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:52:38 smithi146 bash[13136]: debug 2021-03-09T10:52:38.644+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33703 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:52:43.650 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:52:43.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338019650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:43.651 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338019650 secure :-1 s=READY pgs=11524 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:43.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338019650 secure :-1 s=READY pgs=11524 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344003f90).stop 2021-03-09T10:52:43.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338019650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:43.652 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338019650 unknown :-1 s=CLOSED pgs=11524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:43.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:43.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:43.653 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:43.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006e60 2021-03-09T10:52:43.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338062380 con 0x7f833805e5b0 2021-03-09T10:52:43.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073f20 con 0x7f8324006600 2021-03-09T10:52:43.654 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:52:43.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:43.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:52:43.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:43.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:43.655 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:43.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:43.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:43.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:43.656 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338009fc0 con 0x7f833805e5b0 2021-03-09T10:52:43.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.652+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11532 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:43.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c022170 con 0x7f833805e5b0 2021-03-09T10:52:43.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c036340 con 0x7f833805e5b0 2021-03-09T10:52:43.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:43.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833805e5b0 2021-03-09T10:52:44.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:52:43 smithi146 bash[13136]: debug 2021-03-09T10:52:43.652+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33729 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:52:48.657 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:52:48.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:48.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11532 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:48.658 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11532 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c02ed20).stop 2021-03-09T10:52:48.660 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:48.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=CLOSED pgs=11532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:48.661 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:48.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:48.662 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:48.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017d70 con 0x7f8324006e60 2021-03-09T10:52:48.663 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a370 con 0x7f83380292c0 2021-03-09T10:52:48.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324006600 2021-03-09T10:52:48.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:52:48.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:48.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:52:48.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:48.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:48.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:48.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:48.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:48.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:48.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278f0 con 0x7f83380292c0 2021-03-09T10:52:48.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.656+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802a9d0 secure :-1 s=READY pgs=11541 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:48.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.660+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa080 con 0x7f83380292c0 2021-03-09T10:52:48.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.664+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c010560 con 0x7f83380292c0 2021-03-09T10:52:48.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:48.664+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f83380292c0 2021-03-09T10:52:49.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:52:48 smithi146 bash[13136]: debug 2021-03-09T10:52:48.660+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33756 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:52:53.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833802a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:52:53.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833802a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:53.664 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802a9d0 secure :-1 s=READY pgs=11541 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:53.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802a9d0 secure :-1 s=READY pgs=11541 cs=0 l=1 rev1=1 rx=0x7f833c000f20 tx=0x7f833c02ed20).stop 2021-03-09T10:52:53.665 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833802a9d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:53.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802a9d0 unknown :-1 s=CLOSED pgs=11541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:53.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:53.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:53.666 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:53.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006e60 2021-03-09T10:52:53.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017d70 con 0x7f833805e5b0 2021-03-09T10:52:53.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024620 con 0x7f8324006600 2021-03-09T10:52:53.667 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:52:53.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:53.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:52:53.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:53.668 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:53.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:53.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:53.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:53.669 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:53.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338009fc0 con 0x7f833805e5b0 2021-03-09T10:52:53.670 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.663+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11550 cs=0 l=1 rev1=1 rx=0x7f833c029d00 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:53.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.667+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0218d0 con 0x7f833805e5b0 2021-03-09T10:52:53.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079630 con 0x7f833805e5b0 2021-03-09T10:52:53.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:53.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0648b0 con 0x7f833805e5b0 2021-03-09T10:52:54.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:52:53 smithi146 bash[13136]: debug 2021-03-09T10:52:53.668+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33782 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:52:58.671 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:52:58.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:58.672 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11550 cs=0 l=1 rev1=1 rx=0x7f833c029d00 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:58.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11550 cs=0 l=1 rev1=1 rx=0x7f833c029d00 tx=0x7f833c02ed20).stop 2021-03-09T10:52:58.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:58.673 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=CLOSED pgs=11550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:58.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:58.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:58.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:58.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324006e60 2021-03-09T10:52:58.674 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d610 con 0x7f83380292c0 2021-03-09T10:52:58.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006600 2021-03-09T10:52:58.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:52:58.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:58.675 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:58.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:52:58.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.671+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:58.676 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.675+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:58.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.675+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:58.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.675+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:58.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.675+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:58.677 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.675+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338009fc0 con 0x7f83380292c0 2021-03-09T10:52:58.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.675+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11558 cs=0 l=1 rev1=1 rx=0x7f8344042e30 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:58.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404e110 con 0x7f83380292c0 2021-03-09T10:52:58.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f83380292c0 2021-03-09T10:52:58.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:52:58.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f83380292c0 2021-03-09T10:52:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:52:58 smithi146 bash[13136]: debug 2021-03-09T10:52:58.675+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33809 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:53:03.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.675+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:53:03.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.675+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:03.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.675+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11558 cs=0 l=1 rev1=1 rx=0x7f8344042e30 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:03.678 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.675+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11558 cs=0 l=1 rev1=1 rx=0x7f8344042e30 tx=0x7f8344003f90).stop 2021-03-09T10:53:03.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:03.679 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=CLOSED pgs=11558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:03.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:03.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:03.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:03.680 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017d70 con 0x7f8324006e60 2021-03-09T10:53:03.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380093b0 con 0x7f833805e5b0 2021-03-09T10:53:03.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:53:03.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380241f0 con 0x7f8324006600 2021-03-09T10:53:03.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:03.681 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:53:03.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:03.682 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:03.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:03.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:03.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:03.683 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:03.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.679+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800b470 con 0x7f833805e5b0 2021-03-09T10:53:03.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.683+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11570 cs=0 l=1 rev1=1 rx=0x7f8338015d00 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:03.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344014650 con 0x7f833805e5b0 2021-03-09T10:53:03.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.687+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344014650 con 0x7f833805e5b0 2021-03-09T10:53:03.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:03.691+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344014050 con 0x7f833805e5b0 2021-03-09T10:53:04.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:53:03 smithi146 bash[13136]: debug 2021-03-09T10:53:03.679+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33835 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:53:08.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.683+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:53:08.684 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.683+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:08.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.683+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11570 cs=0 l=1 rev1=1 rx=0x7f8338015d00 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:08.685 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.683+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11570 cs=0 l=1 rev1=1 rx=0x7f8338015d00 tx=0x7f8344003f90).stop 2021-03-09T10:53:08.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:08.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.683+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=CLOSED pgs=11570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:08.686 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.683+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:08.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.683+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:08.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.683+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:08.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f8324006e60 2021-03-09T10:53:08.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017d70 con 0x7f83380292c0 2021-03-09T10:53:08.687 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.683+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022270 con 0x7f8324006600 2021-03-09T10:53:08.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.683+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:53:08.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.683+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:08.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.687+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:53:08.688 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.687+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:08.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.687+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:08.689 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.687+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:08.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.687+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:08.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.687+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:08.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.687+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:08.690 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.687+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801dfa0 con 0x7f83380292c0 2021-03-09T10:53:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.687+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11578 cs=0 l=1 rev1=1 rx=0x7f833c04d3f0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.691+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c02ed90 con 0x7f83380292c0 2021-03-09T10:53:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.695+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c009180 con 0x7f83380292c0 2021-03-09T10:53:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:08.695+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c042ce0 con 0x7f83380292c0 2021-03-09T10:53:09.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:53:08 smithi146 bash[13136]: debug 2021-03-09T10:53:08.687+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33861 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:53:13.691 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:53:13.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:13.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11578 cs=0 l=1 rev1=1 rx=0x7f833c04d3f0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:13.693 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11578 cs=0 l=1 rev1=1 rx=0x7f833c04d3f0 tx=0x7f833c02ed20).stop 2021-03-09T10:53:13.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:13.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=CLOSED pgs=11578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:13.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:13.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:13.694 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:13.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380127a0 con 0x7f8324006e60 2021-03-09T10:53:13.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ea60 con 0x7f833805e5b0 2021-03-09T10:53:13.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068fa0 con 0x7f8324006600 2021-03-09T10:53:13.695 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:53:13.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:13.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:53:13.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:13.696 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.691+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:13.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.695+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:13.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.695+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:13.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.695+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:13.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.695+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:13.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.695+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338071df0 con 0x7f833805e5b0 2021-03-09T10:53:13.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.695+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11584 cs=0 l=1 rev1=1 rx=0x7f8344019750 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:13.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.699+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400d450 con 0x7f833805e5b0 2021-03-09T10:53:13.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.699+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405ea90 con 0x7f833805e5b0 2021-03-09T10:53:13.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:13.703+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834405eca0 con 0x7f833805e5b0 2021-03-09T10:53:14.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:53:13 smithi146 bash[13136]: debug 2021-03-09T10:53:13.695+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33888 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:53:18.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.695+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:53:18.697 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.695+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:18.698 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.695+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11584 cs=0 l=1 rev1=1 rx=0x7f8344019750 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:18.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.695+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11584 cs=0 l=1 rev1=1 rx=0x7f8344019750 tx=0x7f8344003f90).stop 2021-03-09T10:53:18.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.695+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:18.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.695+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=CLOSED pgs=11584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:18.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.695+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:18.699 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.695+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:18.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.695+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:18.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.695+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380093b0 con 0x7f8324006e60 2021-03-09T10:53:18.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.695+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801a580 con 0x7f83380292c0 2021-03-09T10:53:18.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.695+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380241f0 con 0x7f8324006600 2021-03-09T10:53:18.700 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.695+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:53:18.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.695+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:18.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.695+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:53:18.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.699+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:18.701 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.699+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:18.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.699+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:18.702 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.699+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:18.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.699+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:18.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.699+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:18.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.699+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380639b0 con 0x7f83380292c0 2021-03-09T10:53:18.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.699+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11594 cs=0 l=1 rev1=1 rx=0x7f834404ed30 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:18.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.703+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440471d0 con 0x7f83380292c0 2021-03-09T10:53:18.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.703+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440471d0 con 0x7f83380292c0 2021-03-09T10:53:18.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:18.707+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402dd90 con 0x7f83380292c0 2021-03-09T10:53:19.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:53:18 smithi146 bash[13136]: debug 2021-03-09T10:53:18.699+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33914 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:53:23.703 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:53:23.704 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:23.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11594 cs=0 l=1 rev1=1 rx=0x7f834404ed30 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:23.705 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 secure :-1 s=READY pgs=11594 cs=0 l=1 rev1=1 rx=0x7f834404ed30 tx=0x7f8344003f90).stop 2021-03-09T10:53:23.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800c560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:23.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800c560 unknown :-1 s=CLOSED pgs=11594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:23.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:23.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:23.706 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:23.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006e60 2021-03-09T10:53:23.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065490 con 0x7f833805e5b0 2021-03-09T10:53:23.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f8324006600 2021-03-09T10:53:23.707 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:53:23.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:23.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:53:23.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:23.708 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:23.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:23.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:23.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:23.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:23.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.703+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017200 con 0x7f833805e5b0 2021-03-09T10:53:23.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.707+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11602 cs=0 l=1 rev1=1 rx=0x7f833c01d6e0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:23.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.711+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f833805e5b0 2021-03-09T10:53:23.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.711+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c026b70 con 0x7f833805e5b0 2021-03-09T10:53:23.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:23.711+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c01d3a0 con 0x7f833805e5b0 2021-03-09T10:53:24.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:53:23 smithi146 bash[13136]: debug 2021-03-09T10:53:23.707+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33941 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:53:28.709 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.707+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:53:28.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.707+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:28.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.707+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11602 cs=0 l=1 rev1=1 rx=0x7f833c01d6e0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:28.710 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.707+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11602 cs=0 l=1 rev1=1 rx=0x7f833c01d6e0 tx=0x7f833c02ed20).stop 2021-03-09T10:53:28.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:28.711 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=CLOSED pgs=11602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:28.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338025380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:28.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:28.712 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:28.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338023a70 con 0x7f8324006e60 2021-03-09T10:53:28.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063dd0 con 0x7f83380292c0 2021-03-09T10:53:28.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380241f0 con 0x7f8324006600 2021-03-09T10:53:28.713 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:53:28.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:53:28.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:28.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:28.714 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338025380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:28.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:28.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:28.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:28.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:28.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.711+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380639b0 con 0x7f83380292c0 2021-03-09T10:53:28.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.715+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338025380 secure :-1 s=READY pgs=11613 cs=0 l=1 rev1=1 rx=0x7f833c05ca60 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:28.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.715+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f83380292c0 2021-03-09T10:53:28.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.715+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f83380292c0 2021-03-09T10:53:28.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:28.723+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c00c150 con 0x7f83380292c0 2021-03-09T10:53:29.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:53:28 smithi146 bash[13136]: debug 2021-03-09T10:53:28.711+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33967 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:53:33.715 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.715+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338025380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:53:33.716 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.715+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338025380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:33.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.715+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338025380 secure :-1 s=READY pgs=11613 cs=0 l=1 rev1=1 rx=0x7f833c05ca60 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:33.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.715+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338025380 secure :-1 s=READY pgs=11613 cs=0 l=1 rev1=1 rx=0x7f833c05ca60 tx=0x7f833c02ed20).stop 2021-03-09T10:53:33.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.715+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338025380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:33.717 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.715+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338025380 unknown :-1 s=CLOSED pgs=11613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:33.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.715+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:33.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.715+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:33.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.715+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:33.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.715+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006e60 2021-03-09T10:53:33.718 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.715+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ed90 con 0x7f833805e5b0 2021-03-09T10:53:33.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.715+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f8324006600 2021-03-09T10:53:33.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.715+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:53:33.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.715+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:33.719 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.715+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:53:33.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.715+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:33.720 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.719+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:33.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.719+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:33.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.719+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:33.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.719+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:33.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.719+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:33.721 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.719+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801ccc0 con 0x7f833805e5b0 2021-03-09T10:53:33.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.719+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11619 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:33.722 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.723+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f833805e5b0 2021-03-09T10:53:33.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.723+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f833805e5b0 2021-03-09T10:53:33.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:33.727+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0648b0 con 0x7f833805e5b0 2021-03-09T10:53:34.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:53:33 smithi146 bash[13136]: debug 2021-03-09T10:53:33.719+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 33994 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:53:38.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:53:38.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:38.723 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11619 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:38.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 secure :-1 s=READY pgs=11619 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c02ed20).stop 2021-03-09T10:53:38.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:38.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833805e5b0 0x7f8338071760 unknown :-1 s=CLOSED pgs=11619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:38.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:38.724 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:38.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:38.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022270 con 0x7f8324006e60 2021-03-09T10:53:38.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ed90 con 0x7f83380292c0 2021-03-09T10:53:38.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024810 con 0x7f8324006600 2021-03-09T10:53:38.725 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:53:38.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:53:38.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:38.726 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:38.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:38.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:38.727 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:38.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:38.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:38.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.722+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338071df0 con 0x7f83380292c0 2021-03-09T10:53:38.728 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.726+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338021080 secure :-1 s=READY pgs=11628 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:38.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.726+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c05ce00 con 0x7f83380292c0 2021-03-09T10:53:38.729 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.726+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c092380 con 0x7f83380292c0 2021-03-09T10:53:38.732 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:38.730+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0a1320 con 0x7f83380292c0 2021-03-09T10:53:39.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:53:38 smithi146 bash[13136]: debug 2021-03-09T10:53:38.723+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34020 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:53:43.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.726+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:53:43.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.726+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:43.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.726+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338021080 secure :-1 s=READY pgs=11628 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:43.733 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.726+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338021080 secure :-1 s=READY pgs=11628 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c02ed20).stop 2021-03-09T10:53:43.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.726+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:43.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.726+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338021080 unknown :-1 s=CLOSED pgs=11628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:43.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.726+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:43.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.726+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338021990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:43.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.726+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:43.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.726+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801cd10 con 0x7f8324006e60 2021-03-09T10:53:43.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.726+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024620 con 0x7f8338021570 2021-03-09T10:53:43.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.726+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380241f0 con 0x7f8324006600 2021-03-09T10:53:43.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.726+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:53:43.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.726+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:43.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.726+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:53:43.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.726+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:43.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.730+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338021990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:43.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.730+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:43.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.730+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:43.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.730+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:43.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.730+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:43.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.730+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801f3c0 con 0x7f8338021570 2021-03-09T10:53:43.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.730+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338021990 secure :-1 s=READY pgs=11638 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344003f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:43.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338021570 2021-03-09T10:53:43.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019af0 con 0x7f8338021570 2021-03-09T10:53:43.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:43.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338021570 2021-03-09T10:53:44.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:53:43 smithi146 bash[13136]: debug 2021-03-09T10:53:43.731+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34047 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:53:48.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338021990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:53:48.734 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338021990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:48.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338021990 secure :-1 s=READY pgs=11638 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344003f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:48.735 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338021990 secure :-1 s=READY pgs=11638 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344003f90).stop 2021-03-09T10:53:48.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338021990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:48.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338021990 unknown :-1 s=CLOSED pgs=11638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:48.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:48.736 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338011970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:48.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:48.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024620 con 0x7f8324006e60 2021-03-09T10:53:48.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063960 con 0x7f83380292c0 2021-03-09T10:53:48.737 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324006600 2021-03-09T10:53:48.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:53:48.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:48.738 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:53:48.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:48.739 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.734+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338011970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:48.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.738+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:48.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.738+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:48.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.738+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:48.740 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.738+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:48.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.738+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017200 con 0x7f83380292c0 2021-03-09T10:53:48.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.738+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338011970 secure :-1 s=READY pgs=11650 cs=0 l=1 rev1=1 rx=0x7f834400e240 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:48.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404aa60 con 0x7f83380292c0 2021-03-09T10:53:48.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405bd70 con 0x7f83380292c0 2021-03-09T10:53:48.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:48.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404e180 con 0x7f83380292c0 2021-03-09T10:53:49.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:53:48 smithi146 bash[13136]: debug 2021-03-09T10:53:48.739+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34073 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:53:53.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.738+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338011970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:53:53.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.738+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338011970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:53.741 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.738+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338011970 secure :-1 s=READY pgs=11650 cs=0 l=1 rev1=1 rx=0x7f834400e240 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:53.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.738+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338011970 secure :-1 s=READY pgs=11650 cs=0 l=1 rev1=1 rx=0x7f834400e240 tx=0x7f8344007620).stop 2021-03-09T10:53:53.742 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.738+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338011970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:53.743 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.738+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338011970 unknown :-1 s=CLOSED pgs=11650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:53.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.742+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:53.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.742+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:53.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.742+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:53.744 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d960 con 0x7f8324006e60 2021-03-09T10:53:53.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380241f0 con 0x7f8338021570 2021-03-09T10:53:53.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.742+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380093b0 con 0x7f8324006600 2021-03-09T10:53:53.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.742+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:53:53.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.742+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:53:53.745 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.742+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:53.746 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.742+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:53.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.742+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:53.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.742+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:53.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.742+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:53.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.742+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:53.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.742+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:53.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.742+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801dfc0 con 0x7f8338021570 2021-03-09T10:53:53.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.746+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11659 cs=0 l=1 rev1=1 rx=0x7f833c0a8680 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:53.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0546e0 con 0x7f8338021570 2021-03-09T10:53:53.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0546e0 con 0x7f8338021570 2021-03-09T10:53:53.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:53.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0648b0 con 0x7f8338021570 2021-03-09T10:53:54.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:53:53 smithi146 bash[13136]: debug 2021-03-09T10:53:53.743+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34100 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:53:58.747 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:53:58.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:58.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11659 cs=0 l=1 rev1=1 rx=0x7f833c0a8680 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:58.748 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11659 cs=0 l=1 rev1=1 rx=0x7f833c0a8680 tx=0x7f833c02ed20).stop 2021-03-09T10:53:58.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:58.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=CLOSED pgs=11659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:58.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:58.749 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:58.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:58.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324006e60 2021-03-09T10:53:58.750 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380241f0 con 0x7f83380292c0 2021-03-09T10:53:58.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380166e0 con 0x7f8324006600 2021-03-09T10:53:58.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:53:58.751 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:58.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:53:58.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:58.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:58.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:58.752 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.746+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800cdf0 con 0x7f83380292c0 2021-03-09T10:53:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.750+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 secure :-1 s=READY pgs=11662 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.754+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344014c70 con 0x7f83380292c0 2021-03-09T10:53:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.754+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019af0 con 0x7f83380292c0 2021-03-09T10:53:58.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:53:58.758+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f83380292c0 2021-03-09T10:53:59.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:53:58 smithi146 bash[13136]: debug 2021-03-09T10:53:58.751+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34126 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:54:03.753 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.750+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:54:03.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.750+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:03.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.750+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 secure :-1 s=READY pgs=11662 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:03.754 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.750+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 secure :-1 s=READY pgs=11662 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344007620).stop 2021-03-09T10:54:03.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.750+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338069320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:03.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.750+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 unknown :-1 s=CLOSED pgs=11662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:03.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.754+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:03.755 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.754+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:03.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.754+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:03.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.754+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380097f0 con 0x7f8324006e60 2021-03-09T10:54:03.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.754+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380241f0 con 0x7f8338021570 2021-03-09T10:54:03.756 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.754+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065f00 con 0x7f8324006600 2021-03-09T10:54:03.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.754+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:54:03.757 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.754+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:03.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.754+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:54:03.758 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.754+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:03.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.754+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:03.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.758+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:03.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.758+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:03.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.758+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:03.759 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.758+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:03.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.758+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338063910 con 0x7f8338021570 2021-03-09T10:54:03.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.758+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11672 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:03.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.758+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c079630 con 0x7f8338021570 2021-03-09T10:54:03.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.762+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079630 con 0x7f8338021570 2021-03-09T10:54:03.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:03.762+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338021570 2021-03-09T10:54:04.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:54:03 smithi146 bash[13136]: debug 2021-03-09T10:54:03.754+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34153 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:54:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.758+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:54:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.758+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.758+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11672 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.758+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11672 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c02ed20).stop 2021-03-09T10:54:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.758+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.758+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=CLOSED pgs=11672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.758+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.758+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.758+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.758+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801cd10 con 0x7f8324006e60 2021-03-09T10:54:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.758+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380093b0 con 0x7f83380292c0 2021-03-09T10:54:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.758+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800bec0 con 0x7f8324006600 2021-03-09T10:54:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.758+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:54:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.762+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.762+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:54:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.762+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.762+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:08.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.762+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:08.765 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.762+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:08.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.762+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:08.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.762+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:08.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.762+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020ad0 con 0x7f83380292c0 2021-03-09T10:54:08.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.762+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 secure :-1 s=READY pgs=11683 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:08.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.766+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c042bb0 con 0x7f83380292c0 2021-03-09T10:54:08.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.770+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0429d0 con 0x7f83380292c0 2021-03-09T10:54:08.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:08.770+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0792c0 con 0x7f83380292c0 2021-03-09T10:54:09.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:54:08 smithi146 bash[13136]: debug 2021-03-09T10:54:08.762+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34178 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:54:13.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:54:13.766 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:13.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 secure :-1 s=READY pgs=11683 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:13.767 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 secure :-1 s=READY pgs=11683 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c02ed20).stop 2021-03-09T10:54:13.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338069320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:13.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 unknown :-1 s=CLOSED pgs=11683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:13.768 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:13.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:13.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338026730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:13.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324006e60 2021-03-09T10:54:13.769 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8338021570 2021-03-09T10:54:13.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ed20 con 0x7f8324006600 2021-03-09T10:54:13.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:54:13.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:13.770 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:54:13.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:13.771 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338026730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:13.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:13.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:13.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:13.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:13.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.766+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801a860 con 0x7f8338021570 2021-03-09T10:54:13.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.770+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338026730 secure :-1 s=READY pgs=11689 cs=0 l=1 rev1=1 rx=0x7f833c042f70 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:13.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.774+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01dbd0 con 0x7f8338021570 2021-03-09T10:54:13.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.774+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01d9f0 con 0x7f8338021570 2021-03-09T10:54:13.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:13.778+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f8338021570 2021-03-09T10:54:14.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:54:13 smithi146 bash[13136]: debug 2021-03-09T10:54:13.770+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34205 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:54:18.772 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.769+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338026730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:54:18.773 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.769+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338026730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:18.774 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.769+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338026730 secure :-1 s=READY pgs=11689 cs=0 l=1 rev1=1 rx=0x7f833c042f70 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:18.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.769+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338026730 secure :-1 s=READY pgs=11689 cs=0 l=1 rev1=1 rx=0x7f833c042f70 tx=0x7f833c02ed20).stop 2021-03-09T10:54:18.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.769+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338026730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:18.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.769+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338026730 unknown :-1 s=CLOSED pgs=11689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:18.775 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.769+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:18.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.769+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:18.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.769+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:18.776 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.769+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d080 con 0x7f8324006e60 2021-03-09T10:54:18.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.769+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dee0 con 0x7f83380292c0 2021-03-09T10:54:18.777 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.769+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324006600 2021-03-09T10:54:18.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.769+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:54:18.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.769+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:18.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.769+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:54:18.778 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.769+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:18.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.773+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:18.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.773+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:18.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.773+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:18.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.773+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:18.779 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.773+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:18.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.773+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278f0 con 0x7f83380292c0 2021-03-09T10:54:18.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.773+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 secure :-1 s=READY pgs=11697 cs=0 l=1 rev1=1 rx=0x7f833c03e2e0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:18.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.777+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c010350 con 0x7f83380292c0 2021-03-09T10:54:18.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.777+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c036340 con 0x7f83380292c0 2021-03-09T10:54:18.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:18.781+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02fc90 con 0x7f83380292c0 2021-03-09T10:54:19.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:54:18 smithi146 bash[13136]: debug 2021-03-09T10:54:18.774+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34231 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:54:23.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:54:23.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:23.780 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 secure :-1 s=READY pgs=11697 cs=0 l=1 rev1=1 rx=0x7f833c03e2e0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:23.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 secure :-1 s=READY pgs=11697 cs=0 l=1 rev1=1 rx=0x7f833c03e2e0 tx=0x7f833c02ed20).stop 2021-03-09T10:54:23.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338069320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:23.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338069320 unknown :-1 s=CLOSED pgs=11697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:23.781 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:23.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:23.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:23.782 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802a530 con 0x7f8324006e60 2021-03-09T10:54:23.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ed20 con 0x7f8338021570 2021-03-09T10:54:23.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380241f0 con 0x7f8324006600 2021-03-09T10:54:23.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:54:23.783 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:23.784 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:54:23.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:23.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.777+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:23.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.781+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:23.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.781+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:23.785 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.781+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:23.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.781+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:23.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.781+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017200 con 0x7f8338021570 2021-03-09T10:54:23.786 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.781+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11706 cs=0 l=1 rev1=1 rx=0x7f833c01a410 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:23.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.785+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f8338021570 2021-03-09T10:54:23.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.785+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c062c00 con 0x7f8338021570 2021-03-09T10:54:23.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:23.789+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c092d80 con 0x7f8338021570 2021-03-09T10:54:24.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:54:23 smithi146 bash[13136]: debug 2021-03-09T10:54:23.782+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34258 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:54:28.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:54:28.787 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:28.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11706 cs=0 l=1 rev1=1 rx=0x7f833c01a410 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:28.788 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11706 cs=0 l=1 rev1=1 rx=0x7f833c01a410 tx=0x7f833c02ed20).stop 2021-03-09T10:54:28.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:28.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=CLOSED pgs=11706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:28.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338064db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:28.789 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:28.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:54:28.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:28.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:28.790 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d7c0 con 0x7f8324006e60 2021-03-09T10:54:28.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f83380292c0 2021-03-09T10:54:28.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380241f0 con 0x7f8324006600 2021-03-09T10:54:28.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:54:28.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:28.791 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338064db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:28.792 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:28.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:28.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:28.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:28.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.785+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802c300 con 0x7f83380292c0 2021-03-09T10:54:28.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.789+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338064db0 secure :-1 s=READY pgs=11714 cs=0 l=1 rev1=1 rx=0x7f833c01a410 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:28.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.793+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c079630 con 0x7f83380292c0 2021-03-09T10:54:28.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.793+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079800 con 0x7f83380292c0 2021-03-09T10:54:28.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:28.797+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02ed90 con 0x7f83380292c0 2021-03-09T10:54:29.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:54:28 smithi146 bash[13136]: debug 2021-03-09T10:54:28.790+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34284 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:54:33.793 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338064db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:54:33.794 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338064db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:33.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338064db0 secure :-1 s=READY pgs=11714 cs=0 l=1 rev1=1 rx=0x7f833c01a410 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:33.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338064db0 secure :-1 s=READY pgs=11714 cs=0 l=1 rev1=1 rx=0x7f833c01a410 tx=0x7f833c02ed20).stop 2021-03-09T10:54:33.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338064db0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:33.795 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338064db0 unknown :-1 s=CLOSED pgs=11714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:33.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338069b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:33.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:33.796 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:33.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a370 con 0x7f8324006e60 2021-03-09T10:54:33.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8338021570 2021-03-09T10:54:33.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d7c0 con 0x7f8324006600 2021-03-09T10:54:33.797 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:54:33.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:33.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:54:33.798 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:33.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338069b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:33.799 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:33.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:33.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:33.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:33.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802c300 con 0x7f8338021570 2021-03-09T10:54:33.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.793+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338069b80 secure :-1 s=READY pgs=11723 cs=0 l=1 rev1=1 rx=0x7f833c0a14e0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:33.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.797+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00e8b0 con 0x7f8338021570 2021-03-09T10:54:33.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.801+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0a1320 con 0x7f8338021570 2021-03-09T10:54:33.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:33.801+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c02ed90 con 0x7f8338021570 2021-03-09T10:54:34.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:54:33 smithi146 bash[13136]: debug 2021-03-09T10:54:33.794+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34310 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:54:38.800 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338069b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:54:38.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338069b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:38.801 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338069b80 secure :-1 s=READY pgs=11723 cs=0 l=1 rev1=1 rx=0x7f833c0a14e0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:38.802 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338069b80 secure :-1 s=READY pgs=11723 cs=0 l=1 rev1=1 rx=0x7f833c0a14e0 tx=0x7f833c02ed20).stop 2021-03-09T10:54:38.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338069b80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:38.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338069b80 unknown :-1 s=CLOSED pgs=11723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:38.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:38.803 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:38.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:38.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b6e0 con 0x7f8324006e60 2021-03-09T10:54:38.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f83380292c0 2021-03-09T10:54:38.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ed20 con 0x7f8324006600 2021-03-09T10:54:38.804 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:54:38.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:38.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:54:38.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:38.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:38.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:38.805 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:38.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:38.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:38.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380061f0 con 0x7f83380292c0 2021-03-09T10:54:38.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.801+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802a990 secure :-1 s=READY pgs=11732 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:38.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.805+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f83380292c0 2021-03-09T10:54:38.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.805+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f83440038e0 con 0x7f83380292c0 2021-03-09T10:54:38.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:38.809+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f83380292c0 2021-03-09T10:54:39.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:54:38 smithi146 bash[13136]: debug 2021-03-09T10:54:38.802+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34337 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:54:43.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833802a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:54:43.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833802a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:43.806 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802a990 secure :-1 s=READY pgs=11732 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:43.807 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802a990 secure :-1 s=READY pgs=11732 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344007620).stop 2021-03-09T10:54:43.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833802a990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:43.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833802a990 unknown :-1 s=CLOSED pgs=11732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:43.808 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:43.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:43.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:43.809 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b6e0 con 0x7f8324006e60 2021-03-09T10:54:43.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802b4b0 con 0x7f8338021570 2021-03-09T10:54:43.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ed20 con 0x7f8324006600 2021-03-09T10:54:43.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:54:43.810 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:54:43.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:43.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:43.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:43.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:43.811 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:43.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:43.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:43.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800fb60 con 0x7f8338021570 2021-03-09T10:54:43.812 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.805+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11740 cs=0 l=1 rev1=1 rx=0x7f833c092dd0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:43.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.813+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c026380 con 0x7f8338021570 2021-03-09T10:54:43.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.813+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079630 con 0x7f8338021570 2021-03-09T10:54:43.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:43.813+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c029490 con 0x7f8338021570 2021-03-09T10:54:44.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:54:43 smithi146 bash[13136]: debug 2021-03-09T10:54:43.810+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34363 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:54:48.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.809+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:54:48.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.809+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:48.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.809+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11740 cs=0 l=1 rev1=1 rx=0x7f833c092dd0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:48.814 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.809+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11740 cs=0 l=1 rev1=1 rx=0x7f833c092dd0 tx=0x7f833c02ed20).stop 2021-03-09T10:54:48.815 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.809+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:48.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.809+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=CLOSED pgs=11740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:48.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.809+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:48.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.809+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:48.816 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.813+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:48.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.813+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f8324006e60 2021-03-09T10:54:48.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.813+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f83380292c0 2021-03-09T10:54:48.817 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.813+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006600 2021-03-09T10:54:48.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.813+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:54:48.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.813+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:48.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.813+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:54:48.818 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.813+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:48.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.813+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:48.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.813+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:48.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.813+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:48.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.813+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:48.819 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.813+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:48.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.813+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802c300 con 0x7f83380292c0 2021-03-09T10:54:48.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.813+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 secure :-1 s=READY pgs=11749 cs=0 l=1 rev1=1 rx=0x7f833c06b040 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:48.820 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.817+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa080 con 0x7f83380292c0 2021-03-09T10:54:48.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.817+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f83380292c0 2021-03-09T10:54:48.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:48.821+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c029530 con 0x7f83380292c0 2021-03-09T10:54:49.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:54:48 smithi146 bash[13136]: debug 2021-03-09T10:54:48.818+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34390 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:54:53.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380296a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:54:53.821 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380296a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:53.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 secure :-1 s=READY pgs=11749 cs=0 l=1 rev1=1 rx=0x7f833c06b040 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:53.822 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 secure :-1 s=READY pgs=11749 cs=0 l=1 rev1=1 rx=0x7f833c06b040 tx=0x7f833c02ed20).stop 2021-03-09T10:54:53.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380296a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:53.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 unknown :-1 s=CLOSED pgs=11749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:53.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:53.823 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:53.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:53.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b6e0 con 0x7f8324006e60 2021-03-09T10:54:53.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338009a30 con 0x7f8338021570 2021-03-09T10:54:53.824 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324006600 2021-03-09T10:54:53.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:54:53.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:54:53.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:53.825 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:53.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:53.826 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:53.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:53.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:53.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:53.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338011720 con 0x7f8338021570 2021-03-09T10:54:53.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.821+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11759 cs=0 l=1 rev1=1 rx=0x7f833c01c4c0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:53.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.825+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338021570 2021-03-09T10:54:53.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.829+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c01c300 con 0x7f8338021570 2021-03-09T10:54:53.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:53.829+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c029500 con 0x7f8338021570 2021-03-09T10:54:54.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:54:53 smithi146 bash[13136]: debug 2021-03-09T10:54:53.822+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34417 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:54:58.827 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:54:58.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:58.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11759 cs=0 l=1 rev1=1 rx=0x7f833c01c4c0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:58.828 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11759 cs=0 l=1 rev1=1 rx=0x7f833c01c4c0 tx=0x7f833c02ed20).stop 2021-03-09T10:54:58.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:58.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=CLOSED pgs=11759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:58.829 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:58.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:58.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:58.830 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800df00 con 0x7f8324006e60 2021-03-09T10:54:58.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026080 con 0x7f83380292c0 2021-03-09T10:54:58.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b6e0 con 0x7f8324006600 2021-03-09T10:54:58.831 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:54:58.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:54:58.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:58.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:58.832 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:58.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:58.833 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:58.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:58.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:58.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806efe0 con 0x7f83380292c0 2021-03-09T10:54:58.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.829+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 secure :-1 s=READY pgs=11766 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:58.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f83380292c0 2021-03-09T10:54:58.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.833+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079800 con 0x7f83380292c0 2021-03-09T10:54:58.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:54:58.837+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c029490 con 0x7f83380292c0 2021-03-09T10:54:59.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:54:58 smithi146 bash[13136]: debug 2021-03-09T10:54:58.830+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34443 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:55:03.834 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380296a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:55:03.835 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380296a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:03.836 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 secure :-1 s=READY pgs=11766 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:03.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 secure :-1 s=READY pgs=11766 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c02ed20).stop 2021-03-09T10:55:03.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380296a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:03.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 unknown :-1 s=CLOSED pgs=11766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:03.837 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:03.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:03.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:03.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f8324006e60 2021-03-09T10:55:03.838 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806a370 con 0x7f8338021570 2021-03-09T10:55:03.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b6e0 con 0x7f8324006600 2021-03-09T10:55:03.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:55:03.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:03.839 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:55:03.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:03.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:03.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:03.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:03.840 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:03.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:03.841 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.832+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801ac10 con 0x7f8338021570 2021-03-09T10:55:03.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.836+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11775 cs=0 l=1 rev1=1 rx=0x7f833c027fd0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:03.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c054ea0 con 0x7f8338021570 2021-03-09T10:55:03.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c054ea0 con 0x7f8338021570 2021-03-09T10:55:03.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:03.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c041d90 con 0x7f8338021570 2021-03-09T10:55:04.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:55:03 smithi146 bash[13136]: debug 2021-03-09T10:55:03.838+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34469 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:55:08.842 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:55:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11775 cs=0 l=1 rev1=1 rx=0x7f833c027fd0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11775 cs=0 l=1 rev1=1 rx=0x7f833c027fd0 tx=0x7f833c02ed20).stop 2021-03-09T10:55:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=CLOSED pgs=11775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d7b0 con 0x7f8324006e60 2021-03-09T10:55:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338011ed0 con 0x7f83380292c0 2021-03-09T10:55:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006600 2021-03-09T10:55:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:55:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:08.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:08.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:55:08.847 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.840+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:08.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:08.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:08.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:08.848 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:08.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.844+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338004ef0 con 0x7f83380292c0 2021-03-09T10:55:08.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.844+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833806c690 secure :-1 s=READY pgs=11787 cs=0 l=1 rev1=1 rx=0x7f833c026d80 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:08.849 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c079800 con 0x7f83380292c0 2021-03-09T10:55:08.850 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c026df0 con 0x7f83380292c0 2021-03-09T10:55:08.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:08.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c025d00 con 0x7f83380292c0 2021-03-09T10:55:09.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:55:08 smithi146 bash[13136]: debug 2021-03-09T10:55:08.845+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34496 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:55:13.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:55:13.851 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:13.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833806c690 secure :-1 s=READY pgs=11787 cs=0 l=1 rev1=1 rx=0x7f833c026d80 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:13.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833806c690 secure :-1 s=READY pgs=11787 cs=0 l=1 rev1=1 rx=0x7f833c026d80 tx=0x7f833c02ed20).stop 2021-03-09T10:55:13.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833806c690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:13.852 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833806c690 unknown :-1 s=CLOSED pgs=11787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:13.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:13.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:13.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:13.853 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026080 con 0x7f8324006e60 2021-03-09T10:55:13.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b6e0 con 0x7f8338021570 2021-03-09T10:55:13.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024e30 con 0x7f8324006600 2021-03-09T10:55:13.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:55:13.854 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:13.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:55:13.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:13.855 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:13.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:13.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:13.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:13.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:13.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068240 con 0x7f8338021570 2021-03-09T10:55:13.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.848+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338074490 secure :-1 s=READY pgs=11792 cs=0 l=1 rev1=1 rx=0x7f834400e010 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:13.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403c280 con 0x7f8338021570 2021-03-09T10:55:13.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338021570 2021-03-09T10:55:13.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:13.856+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f8338021570 2021-03-09T10:55:14.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:55:13 smithi146 bash[13136]: debug 2021-03-09T10:55:13.849+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34522 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:55:18.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:55:18.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:18.856 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338074490 secure :-1 s=READY pgs=11792 cs=0 l=1 rev1=1 rx=0x7f834400e010 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:18.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338074490 secure :-1 s=READY pgs=11792 cs=0 l=1 rev1=1 rx=0x7f834400e010 tx=0x7f8344007620).stop 2021-03-09T10:55:18.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338074490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:18.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338074490 unknown :-1 s=CLOSED pgs=11792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:18.857 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:18.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:18.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:18.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800ed20 con 0x7f8324006e60 2021-03-09T10:55:18.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800a060 con 0x7f83380292c0 2021-03-09T10:55:18.858 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380079d0 con 0x7f8324006600 2021-03-09T10:55:18.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:55:18.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:18.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:55:18.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:18.859 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:18.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:18.860 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:18.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:18.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:18.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017110 con 0x7f83380292c0 2021-03-09T10:55:18.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.852+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 secure :-1 s=READY pgs=11801 cs=0 l=1 rev1=1 rx=0x7f834400e240 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:18.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.860+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834407d2d0 con 0x7f83380292c0 2021-03-09T10:55:18.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.860+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344017640 con 0x7f83380292c0 2021-03-09T10:55:18.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:18.860+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ac90 con 0x7f83380292c0 2021-03-09T10:55:19.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:55:18 smithi146 bash[13136]: debug 2021-03-09T10:55:18.857+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34549 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:55:23.861 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380296a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:55:23.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380296a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:23.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 secure :-1 s=READY pgs=11801 cs=0 l=1 rev1=1 rx=0x7f834400e240 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:23.862 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 secure :-1 s=READY pgs=11801 cs=0 l=1 rev1=1 rx=0x7f834400e240 tx=0x7f8344007620).stop 2021-03-09T10:55:23.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f83380296a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:23.863 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f83380296a0 unknown :-1 s=CLOSED pgs=11801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:23.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:23.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:23.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:23.864 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c6f0 con 0x7f8324006e60 2021-03-09T10:55:23.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024e30 con 0x7f8338021570 2021-03-09T10:55:23.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801cd10 con 0x7f8324006600 2021-03-09T10:55:23.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:55:23.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:23.865 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:55:23.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:23.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:23.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:23.866 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:23.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:23.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:23.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022e20 con 0x7f8338021570 2021-03-09T10:55:23.867 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.860+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11810 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:23.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.864+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834403c280 con 0x7f8338021570 2021-03-09T10:55:23.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.864+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f8338021570 2021-03-09T10:55:23.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:23.868+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344047080 con 0x7f8338021570 2021-03-09T10:55:24.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:55:23 smithi146 bash[13136]: debug 2021-03-09T10:55:23.861+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34575 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:55:28.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:55:28.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:28.868 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11810 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:28.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11810 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344007620).stop 2021-03-09T10:55:28.869 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:28.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=CLOSED pgs=11810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:28.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:28.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:28.870 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:28.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:55:28.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:28.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338061580 con 0x7f8324006e60 2021-03-09T10:55:28.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063dd0 con 0x7f83380292c0 2021-03-09T10:55:28.871 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b170 con 0x7f8324006600 2021-03-09T10:55:28.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:28.872 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:55:28.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:28.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:28.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:28.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:28.873 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:28.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020ad0 con 0x7f83380292c0 2021-03-09T10:55:28.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.868+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800d380 secure :-1 s=READY pgs=11818 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:28.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.872+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f83380292c0 2021-03-09T10:55:28.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.872+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834403c280 con 0x7f83380292c0 2021-03-09T10:55:28.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:28.876+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834406c750 con 0x7f83380292c0 2021-03-09T10:55:29.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:55:28 smithi146 bash[13136]: debug 2021-03-09T10:55:28.869+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34602 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:55:33.874 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:55:33.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:33.875 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800d380 secure :-1 s=READY pgs=11818 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:33.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800d380 secure :-1 s=READY pgs=11818 cs=0 l=1 rev1=1 rx=0x7f8344049b80 tx=0x7f8344007620).stop 2021-03-09T10:55:33.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f833800d380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:33.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f833800d380 unknown :-1 s=CLOSED pgs=11818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:33.877 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:33.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:55:33.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:33.878 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:33.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338029f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:33.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:55:33.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:33.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b6e0 con 0x7f8324006e60 2021-03-09T10:55:33.879 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338029f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:33.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b170 con 0x7f8338021570 2021-03-09T10:55:33.880 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338017160 con 0x7f8324006600 2021-03-09T10:55:33.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:33.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:33.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:33.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:33.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338010600 con 0x7f8338021570 2021-03-09T10:55:33.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.876+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338029f50 secure :-1 s=READY pgs=11827 cs=0 l=1 rev1=1 rx=0x7f833c01cd40 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:33.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.880+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa080 con 0x7f8338021570 2021-03-09T10:55:33.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.880+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0292c0 con 0x7f8338021570 2021-03-09T10:55:33.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:33.884+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c026df0 con 0x7f8338021570 2021-03-09T10:55:34.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:55:33 smithi146 bash[13136]: debug 2021-03-09T10:55:33.877+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34628 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:55:38.881 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.876+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338029f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:55:38.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.876+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338029f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:38.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.876+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338029f50 secure :-1 s=READY pgs=11827 cs=0 l=1 rev1=1 rx=0x7f833c01cd40 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:38.882 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.876+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338029f50 secure :-1 s=READY pgs=11827 cs=0 l=1 rev1=1 rx=0x7f833c01cd40 tx=0x7f833c02ed20).stop 2021-03-09T10:55:38.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.876+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338029f50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:38.883 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.876+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338029f50 unknown :-1 s=CLOSED pgs=11827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:38.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.876+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:38.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:38.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338072170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:38.884 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ac60 con 0x7f8324006e60 2021-03-09T10:55:38.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f83380292c0 2021-03-09T10:55:38.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060b20 con 0x7f8324006600 2021-03-09T10:55:38.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:55:38.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:38.885 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:55:38.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:38.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338072170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:38.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:38.886 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:38.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 msgr2=0x7f8338072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T10:55:38.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83380292c0 0x7f8338072170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:38.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338007340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:38.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338007340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:38.887 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f83240053d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:38.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 0x7f8324005aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:38.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8342ffd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8324007240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:38.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ac60 con 0x7f8324006e60 2021-03-09T10:55:38.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f8324006a40 2021-03-09T10:55:38.888 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f83240053d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:55:38.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f8342ffd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338060b20 con 0x7f8324006600 2021-03-09T10:55:38.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f83240053d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:38.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8324007240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:55:38.889 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8324007240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:38.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.880+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 0x7f8324005aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:38.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.884+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f83240053d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:38.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.884+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f83240053d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:38.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.884+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8324007240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:38.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.884+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8324007240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:38.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.884+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338011720 con 0x7f8324006a40 2021-03-09T10:55:38.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.884+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 0x7f8324005aa0 secure :-1 s=READY pgs=11840 cs=0 l=1 rev1=1 rx=0x7f8344056b70 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:38.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.884+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344010a00 con 0x7f8324006a40 2021-03-09T10:55:38.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.888+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8324006a40 2021-03-09T10:55:38.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:38.888+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344007740 con 0x7f8324006a40 2021-03-09T10:55:39.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:55:38 smithi146 bash[13136]: debug 2021-03-09T10:55:38.881+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34655 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:55:43.890 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.888+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 msgr2=0x7f8324005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:55:43.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.888+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 msgr2=0x7f8324005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:43.891 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.888+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 0x7f8324005aa0 secure :-1 s=READY pgs=11840 cs=0 l=1 rev1=1 rx=0x7f8344056b70 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:43.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.888+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 0x7f8324005aa0 secure :-1 s=READY pgs=11840 cs=0 l=1 rev1=1 rx=0x7f8344056b70 tx=0x7f8344007620).stop 2021-03-09T10:55:43.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.888+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 msgr2=0x7f8324005aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:43.892 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.888+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 0x7f8324005aa0 unknown :-1 s=CLOSED pgs=11840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:43.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.888+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:43.893 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.888+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:43.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.888+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338007340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:43.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.888+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dee0 con 0x7f8324006e60 2021-03-09T10:55:43.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.888+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380093b0 con 0x7f8338021570 2021-03-09T10:55:43.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.888+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802c4a0 con 0x7f8324006600 2021-03-09T10:55:43.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.892+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:55:43.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.892+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:43.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.892+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338007340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:43.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.892+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:55:43.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.892+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:43.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.896+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:43.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.896+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:43.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.896+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:43.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.896+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:43.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.896+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338018b80 con 0x7f8338021570 2021-03-09T10:55:43.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.896+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338007340 secure :-1 s=READY pgs=11849 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:43.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.896+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338021570 2021-03-09T10:55:43.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.896+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c036340 con 0x7f8338021570 2021-03-09T10:55:43.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:43.896+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338021570 2021-03-09T10:55:44.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:55:43 smithi146 bash[13136]: debug 2021-03-09T10:55:43.889+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34681 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:55:48.895 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338007340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:55:48.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338007340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:48.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338007340 secure :-1 s=READY pgs=11849 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:48.896 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338007340 secure :-1 s=READY pgs=11849 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c02ed20).stop 2021-03-09T10:55:48.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338007340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:48.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338007340 unknown :-1 s=CLOSED pgs=11849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:48.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:48.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 0x7f8338021950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:48.897 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:48.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b170 con 0x7f8324006e60 2021-03-09T10:55:48.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020290 con 0x7f8324006a40 2021-03-09T10:55:48.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f8324006600 2021-03-09T10:55:48.898 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:55:48.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:48.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:55:48.899 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:48.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 0x7f8338021950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:48.900 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:48.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:48.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:48.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:48.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020ad0 con 0x7f8324006a40 2021-03-09T10:55:48.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 0x7f8338021950 secure :-1 s=READY pgs=11853 cs=0 l=1 rev1=1 rx=0x7f83440667d0 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:48.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8324006a40 2021-03-09T10:55:48.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.895+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8324006a40 2021-03-09T10:55:48.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:48.899+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834403bcd0 con 0x7f8324006a40 2021-03-09T10:55:49.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:55:48 smithi146 bash[13136]: debug 2021-03-09T10:55:48.897+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34708 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:55:53.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.899+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 msgr2=0x7f8338021950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:55:53.901 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.899+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 msgr2=0x7f8338021950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.899+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 0x7f8338021950 secure :-1 s=READY pgs=11853 cs=0 l=1 rev1=1 rx=0x7f83440667d0 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.899+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 0x7f8338021950 secure :-1 s=READY pgs=11853 cs=0 l=1 rev1=1 rx=0x7f83440667d0 tx=0x7f8344007620).stop 2021-03-09T10:55:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.899+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 msgr2=0x7f8338021950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.899+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8324006a40 0x7f8338021950 unknown :-1 s=CLOSED pgs=11853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.899+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.899+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.899+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.899+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021430 con 0x7f8324006e60 2021-03-09T10:55:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.899+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026080 con 0x7f8338021570 2021-03-09T10:55:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.899+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b170 con 0x7f8324006600 2021-03-09T10:55:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.899+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:55:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.899+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.903+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:55:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.903+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.903+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:53.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.903+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:53.906 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.903+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.903+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.903+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.903+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380241a0 con 0x7f8338021570 2021-03-09T10:55:53.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.907+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11865 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:53.908 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.907+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344056db0 con 0x7f8338021570 2021-03-09T10:55:53.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.911+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344013770 con 0x7f8338021570 2021-03-09T10:55:53.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:53.911+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404e110 con 0x7f8338021570 2021-03-09T10:55:54.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:55:53 smithi146 bash[13136]: debug 2021-03-09T10:55:53.901+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34734 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:55:58.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:55:58.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:58.909 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11865 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:58.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11865 cs=0 l=1 rev1=1 rx=0x7f834404c190 tx=0x7f8344007620).stop 2021-03-09T10:55:58.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:58.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=CLOSED pgs=11865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:58.910 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324006e60 2021-03-09T10:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f833800a5f0 2021-03-09T10:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f8324006600 2021-03-09T10:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:55:58.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:58.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:55:58.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:58.912 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.907+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:58.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.911+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:58.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.911+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:58.913 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.911+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:58.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.911+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:58.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.911+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380241a0 con 0x7f833800a5f0 2021-03-09T10:55:58.914 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.911+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800aa30 secure :-1 s=READY pgs=11870 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:58.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.915+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402b400 con 0x7f833800a5f0 2021-03-09T10:55:58.916 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.915+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402b220 con 0x7f833800a5f0 2021-03-09T10:55:58.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:55:58.919+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833800a5f0 2021-03-09T10:55:59.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:55:58 smithi146 bash[13136]: debug 2021-03-09T10:55:58.909+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34761 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:56:03.917 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833800aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:56:03.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833800aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:03.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800aa30 secure :-1 s=READY pgs=11870 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:03.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800aa30 secure :-1 s=READY pgs=11870 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344007620).stop 2021-03-09T10:56:03.918 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833800aa30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:03.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800aa30 unknown :-1 s=CLOSED pgs=11870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:03.919 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:03.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:03.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:03.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024060 con 0x7f8324006e60 2021-03-09T10:56:03.920 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020290 con 0x7f8338021570 2021-03-09T10:56:03.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b170 con 0x7f8324006600 2021-03-09T10:56:03.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:56:03.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:03.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:56:03.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:03.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.915+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:03.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.919+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:03.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.919+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:03.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.919+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:03.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.919+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:03.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.919+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380105d0 con 0x7f8338021570 2021-03-09T10:56:03.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.919+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11881 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:03.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.923+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa080 con 0x7f8338021570 2021-03-09T10:56:03.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.923+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c036340 con 0x7f8338021570 2021-03-09T10:56:03.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:03.923+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f8338021570 2021-03-09T10:56:04.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:56:03 smithi146 bash[13136]: debug 2021-03-09T10:56:03.917+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34787 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:56:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.919+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:56:08.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.919+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:08.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.919+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11881 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:08.922 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.919+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11881 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c02ed20).stop 2021-03-09T10:56:08.923 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.919+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:08.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.919+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=CLOSED pgs=11881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:08.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.919+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:08.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.919+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:08.924 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.919+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:08.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.919+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802da90 con 0x7f8324006e60 2021-03-09T10:56:08.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.919+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f833800a5f0 2021-03-09T10:56:08.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.919+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063280 con 0x7f8324006600 2021-03-09T10:56:08.925 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.919+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:56:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.919+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.923+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:56:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.923+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.923+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:08.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.923+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:08.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.923+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:08.927 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.923+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:08.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.923+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:08.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.923+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338011030 con 0x7f833800a5f0 2021-03-09T10:56:08.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.923+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800aa30 secure :-1 s=READY pgs=11888 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:08.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.927+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83440112f0 con 0x7f833800a5f0 2021-03-09T10:56:08.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.927+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834407c630 con 0x7f833800a5f0 2021-03-09T10:56:08.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:08.931+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402c2c0 con 0x7f833800a5f0 2021-03-09T10:56:09.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:56:08 smithi146 bash[13136]: debug 2021-03-09T10:56:08.920+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34814 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:56:13.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833800aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:56:13.928 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833800aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:13.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800aa30 secure :-1 s=READY pgs=11888 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:13.929 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800aa30 secure :-1 s=READY pgs=11888 cs=0 l=1 rev1=1 rx=0x7f834400b330 tx=0x7f8344007620).stop 2021-03-09T10:56:13.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833800aa30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:13.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800aa30 unknown :-1 s=CLOSED pgs=11888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:13.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:13.930 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:13.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:13.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802a6e0 con 0x7f8324006e60 2021-03-09T10:56:13.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b170 con 0x7f8338021570 2021-03-09T10:56:13.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064850 con 0x7f8324006600 2021-03-09T10:56:13.931 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:56:13.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:13.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:13.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:56:13.932 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.927+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:13.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.931+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:13.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.931+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:13.933 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.931+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:13.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.931+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:13.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.931+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024620 con 0x7f8338021570 2021-03-09T10:56:13.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.931+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11896 cs=0 l=1 rev1=1 rx=0x7f833c0a14c0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:13.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c036340 con 0x7f8338021570 2021-03-09T10:56:13.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079800 con 0x7f8338021570 2021-03-09T10:56:13.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:13.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c026dc0 con 0x7f8338021570 2021-03-09T10:56:14.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:56:13 smithi146 bash[13136]: debug 2021-03-09T10:56:13.928+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34840 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:56:18.934 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:56:18.935 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11896 cs=0 l=1 rev1=1 rx=0x7f833c0a14c0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11896 cs=0 l=1 rev1=1 rx=0x7f833c0a14c0 tx=0x7f833c02ed20).stop 2021-03-09T10:56:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=CLOSED pgs=11896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802ab90 con 0x7f8324006e60 2021-03-09T10:56:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b890 con 0x7f833800a5f0 2021-03-09T10:56:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020580 con 0x7f8324006600 2021-03-09T10:56:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:56:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:56:18.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:18.939 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:18.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:18.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:18.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:18.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:18.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.935+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024620 con 0x7f833800a5f0 2021-03-09T10:56:18.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.939+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800a9d0 secure :-1 s=READY pgs=11905 cs=0 l=1 rev1=1 rx=0x7f8344012000 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:18.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400ded0 con 0x7f833800a5f0 2021-03-09T10:56:18.942 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400ded0 con 0x7f833800a5f0 2021-03-09T10:56:18.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:18.943+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344067320 con 0x7f833800a5f0 2021-03-09T10:56:19.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:56:18 smithi146 bash[13136]: debug 2021-03-09T10:56:18.936+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34867 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:56:23.940 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.939+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833800a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:56:23.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.939+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833800a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:23.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.939+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800a9d0 secure :-1 s=READY pgs=11905 cs=0 l=1 rev1=1 rx=0x7f8344012000 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:23.941 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.939+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800a9d0 secure :-1 s=READY pgs=11905 cs=0 l=1 rev1=1 rx=0x7f8344012000 tx=0x7f8344007620).stop 2021-03-09T10:56:23.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833800a9d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:23.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.939+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833800a9d0 unknown :-1 s=CLOSED pgs=11905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:23.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.939+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:23.943 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.939+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:23.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.939+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:23.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b170 con 0x7f8324006e60 2021-03-09T10:56:23.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f8338021570 2021-03-09T10:56:23.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.939+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068290 con 0x7f8324006600 2021-03-09T10:56:23.944 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.939+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:56:23.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.939+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:23.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.939+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:56:23.945 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.939+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:23.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.943+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:23.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.943+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:23.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.943+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:23.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.943+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:23.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.943+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:23.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.943+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806d9b0 con 0x7f8338021570 2021-03-09T10:56:23.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.943+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11915 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:23.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.947+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338021570 2021-03-09T10:56:23.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.947+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f8338021570 2021-03-09T10:56:23.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:23.951+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344011a80 con 0x7f8338021570 2021-03-09T10:56:24.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:56:23 smithi146 bash[13136]: debug 2021-03-09T10:56:23.940+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34893 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:56:28.946 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:56:28.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:28.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11915 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:28.947 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11915 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344007620).stop 2021-03-09T10:56:28.948 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:28.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=CLOSED pgs=11915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:28.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:28.949 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:28.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:28.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020130 con 0x7f8324006e60 2021-03-09T10:56:28.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e690 con 0x7f833800a5f0 2021-03-09T10:56:28.950 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b890 con 0x7f8324006600 2021-03-09T10:56:28.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:56:28.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:28.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:56:28.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:28.951 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:28.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:28.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:28.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:28.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:28.952 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.946+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338011030 con 0x7f833800a5f0 2021-03-09T10:56:28.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.950+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833806d850 secure :-1 s=READY pgs=11922 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c02ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:28.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa080 con 0x7f833800a5f0 2021-03-09T10:56:28.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079800 con 0x7f833800a5f0 2021-03-09T10:56:28.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:28.954+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f833800a5f0 2021-03-09T10:56:29.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:56:28 smithi146 bash[13136]: debug 2021-03-09T10:56:28.948+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34919 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:56:33.953 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.950+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833806d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:56:33.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.950+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833806d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:33.954 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.950+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833806d850 secure :-1 s=READY pgs=11922 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c02ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:33.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.950+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833806d850 secure :-1 s=READY pgs=11922 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c02ed20).stop 2021-03-09T10:56:33.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.950+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833806d850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:33.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.950+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833806d850 unknown :-1 s=CLOSED pgs=11922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:33.955 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.950+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:33.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.950+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:33.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.950+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:33.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.950+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020580 con 0x7f8324006e60 2021-03-09T10:56:33.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.950+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b010 con 0x7f8338021570 2021-03-09T10:56:33.956 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.950+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b170 con 0x7f8324006600 2021-03-09T10:56:33.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.950+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:56:33.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.950+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:56:33.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.950+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:33.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.950+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:33.957 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.954+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:33.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.954+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:33.958 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.954+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:33.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.954+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:33.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.954+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:33.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.954+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338066ff0 con 0x7f8338021570 2021-03-09T10:56:33.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.954+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11931 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c06b960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:33.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f8338021570 2021-03-09T10:56:33.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c045600 con 0x7f8338021570 2021-03-09T10:56:33.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:33.962+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0648b0 con 0x7f8338021570 2021-03-09T10:56:34.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:56:33 smithi146 bash[13136]: debug 2021-03-09T10:56:33.952+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34946 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:56:38.959 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:56:38.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:38.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11931 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c06b960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:38.960 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11931 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c06b960).stop 2021-03-09T10:56:38.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:38.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=CLOSED pgs=11931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:38.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:38.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:38.962 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f8338027060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:38.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801ac20 con 0x7f8324006e60 2021-03-09T10:56:38.963 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074940 con 0x7f833800a5f0 2021-03-09T10:56:38.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024670 con 0x7f8324006600 2021-03-09T10:56:38.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:56:38.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:38.964 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:56:38.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:38.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.958+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f8338027060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:38.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.962+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:38.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:38.965 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.962+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:38.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:38.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.962+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017130 con 0x7f833800a5f0 2021-03-09T10:56:38.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.962+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f8338027060 secure :-1 s=READY pgs=11941 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c06b960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:38.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f833800a5f0 2021-03-09T10:56:38.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c042560 con 0x7f833800a5f0 2021-03-09T10:56:38.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:38.970+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c064880 con 0x7f833800a5f0 2021-03-09T10:56:39.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:56:38 smithi146 bash[13136]: debug 2021-03-09T10:56:38.960+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34972 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:56:43.966 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f8338027060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:56:43.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f8338027060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:43.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f8338027060 secure :-1 s=READY pgs=11941 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c06b960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:43.967 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f8338027060 secure :-1 s=READY pgs=11941 cs=0 l=1 rev1=1 rx=0x7f833c000ea0 tx=0x7f833c06b960).stop 2021-03-09T10:56:43.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f8338027060 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:43.968 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f8338027060 unknown :-1 s=CLOSED pgs=11941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:43.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:43.969 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:43.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:43.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020580 con 0x7f8324006e60 2021-03-09T10:56:43.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380083d0 con 0x7f8338021570 2021-03-09T10:56:43.970 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380166e0 con 0x7f8324006600 2021-03-09T10:56:43.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:56:43.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:43.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:43.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:56:43.971 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.966+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:43.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:43.972 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:43.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:43.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:43.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338018b80 con 0x7f8338021570 2021-03-09T10:56:43.973 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11949 cs=0 l=1 rev1=1 rx=0x7f833c041680 tx=0x7f833c06b960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:43.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.974+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa030 con 0x7f8338021570 2021-03-09T10:56:43.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.974+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0416f0 con 0x7f8338021570 2021-03-09T10:56:43.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:43.978+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c004000 con 0x7f8338021570 2021-03-09T10:56:44.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:56:43 smithi146 bash[13136]: debug 2021-03-09T10:56:43.968+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 34999 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:56:48.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:56:48.974 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.970+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:48.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11949 cs=0 l=1 rev1=1 rx=0x7f833c041680 tx=0x7f833c06b960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:48.975 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.970+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 secure :-1 s=READY pgs=11949 cs=0 l=1 rev1=1 rx=0x7f833c041680 tx=0x7f833c06b960).stop 2021-03-09T10:56:48.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:48.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338021570 0x7f8338071760 unknown :-1 s=CLOSED pgs=11949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:48.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f83380727e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:48.976 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:48.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:48.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f8324006e60 2021-03-09T10:56:48.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074940 con 0x7f833800a5f0 2021-03-09T10:56:48.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068290 con 0x7f8324006600 2021-03-09T10:56:48.977 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:56:48.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:48.978 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:56:48.979 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:48.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f83380727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:48.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:48.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:48.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:48.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:48.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338066ff0 con 0x7f833800a5f0 2021-03-09T10:56:48.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.974+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f83380727e0 secure :-1 s=READY pgs=11958 cs=0 l=1 rev1=1 rx=0x7f833c045cf0 tx=0x7f833c06b960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:48.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.978+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f833800a5f0 2021-03-09T10:56:48.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.978+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c010e50 con 0x7f833800a5f0 2021-03-09T10:56:48.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:48.982+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c072390 con 0x7f833800a5f0 2021-03-09T10:56:49.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:56:48 smithi146 bash[13136]: debug 2021-03-09T10:56:48.972+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 35062 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:56:53.980 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f83380727e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:56:53.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f83380727e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:53.981 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f83380727e0 secure :-1 s=READY pgs=11958 cs=0 l=1 rev1=1 rx=0x7f833c045cf0 tx=0x7f833c06b960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:53.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f83380727e0 secure :-1 s=READY pgs=11958 cs=0 l=1 rev1=1 rx=0x7f833c045cf0 tx=0x7f833c06b960).stop 2021-03-09T10:56:53.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f83380727e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:53.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f83380727e0 unknown :-1 s=CLOSED pgs=11958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:53.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:53.982 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:53.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:53.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063ef0 con 0x7f8324006e60 2021-03-09T10:56:53.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338003110 con 0x7f833802c100 2021-03-09T10:56:53.983 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020580 con 0x7f8324006600 2021-03-09T10:56:53.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:56:53.984 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:53.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:56:53.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:53.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.978+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:53.985 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.982+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:53.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.982+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:53.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.982+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:53.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.982+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:53.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.982+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338061530 con 0x7f833802c100 2021-03-09T10:56:53.986 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.982+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=11966 cs=0 l=1 rev1=1 rx=0x7f834404b0c0 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:53.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.986+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8344004520 con 0x7f833802c100 2021-03-09T10:56:53.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.986+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402af00 con 0x7f833802c100 2021-03-09T10:56:53.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:53.990+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833802c100 2021-03-09T10:56:54.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:56:53 smithi146 bash[13136]: debug 2021-03-09T10:56:53.980+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 35124 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:56:58.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.982+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:56:58.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.982+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:58.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.982+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=11966 cs=0 l=1 rev1=1 rx=0x7f834404b0c0 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:58.987 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.986+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=11966 cs=0 l=1 rev1=1 rx=0x7f834404b0c0 tx=0x7f8344007620).stop 2021-03-09T10:56:58.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.986+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:58.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.986+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=CLOSED pgs=11966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:58.988 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.986+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:58.989 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.986+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:58.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.986+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:56:58.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.986+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:58.990 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.986+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:58.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.986+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f8324006e60 2021-03-09T10:56:58.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.986+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:58.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.986+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:56:58.991 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.986+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:58.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.986+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800dfa0 con 0x7f833800a5f0 2021-03-09T10:56:58.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.986+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012840 con 0x7f8324006600 2021-03-09T10:56:58.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.990+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:58.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.990+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:58.992 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.990+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:58.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.990+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:58.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.990+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024620 con 0x7f833800a5f0 2021-03-09T10:56:58.993 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.990+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 secure :-1 s=READY pgs=11976 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:58.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.990+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405bc80 con 0x7f833800a5f0 2021-03-09T10:56:58.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.994+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405bc80 con 0x7f833800a5f0 2021-03-09T10:56:58.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:56:58.994+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404b800 con 0x7f833800a5f0 2021-03-09T10:56:59.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:56:58 smithi146 bash[13136]: debug 2021-03-09T10:56:58.992+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 35184 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:57:03.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.990+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833802c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:57:03.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.990+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833802c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:03.994 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.990+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 secure :-1 s=READY pgs=11976 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:03.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.990+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 secure :-1 s=READY pgs=11976 cs=0 l=1 rev1=1 rx=0x7f83440489a0 tx=0x7f8344007620).stop 2021-03-09T10:57:03.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.990+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833802c4e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:03.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.990+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 unknown :-1 s=CLOSED pgs=11976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:03.995 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.990+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:03.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.990+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:03.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.990+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:57:03.996 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.990+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:03.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.994+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:03.997 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.994+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:03.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.994+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:57:03.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.994+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:03.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.994+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e470 con 0x7f8324006e60 2021-03-09T10:57:03.998 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.994+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e180 con 0x7f833802c100 2021-03-09T10:57:03.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.994+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020580 con 0x7f8324006600 2021-03-09T10:57:03.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.994+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:03.999 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.994+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:04.000 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.994+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:04.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.994+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:04.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.994+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017b80 con 0x7f833802c100 2021-03-09T10:57:04.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.994+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=11986 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:04.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:03.998+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834405bc80 con 0x7f833802c100 2021-03-09T10:57:04.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:04.002+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344019270 con 0x7f833802c100 2021-03-09T10:57:04.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:04.002+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833802c100 2021-03-09T10:57:04.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:57:04 smithi146 bash[13136]: debug 2021-03-09T10:57:04.000+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 35247 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:57:09.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:08.997+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:57:09.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:08.997+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:09.001 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:08.997+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=11986 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:09.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:08.997+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=11986 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344007620).stop 2021-03-09T10:57:09.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:08.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:09.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:08.997+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=CLOSED pgs=11986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:09.002 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:08.997+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:09.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:08.997+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:09.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:08.997+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:09.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:08.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338027940 con 0x7f8324006e60 2021-03-09T10:57:09.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:08.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f833800a5f0 2021-03-09T10:57:09.003 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:08.997+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338073fa0 con 0x7f8324006600 2021-03-09T10:57:09.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:08.997+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:57:09.004 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:08.997+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:09.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:08.997+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:57:09.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:08.997+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:09.005 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:09.001+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:09.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:09.001+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:09.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:09.001+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:09.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:09.001+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:09.006 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:09.001+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:09.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:09.001+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020ad0 con 0x7f833800a5f0 2021-03-09T10:57:09.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:09.001+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 secure :-1 s=READY pgs=11992 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c06b960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:09.007 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:09.005+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c079630 con 0x7f833800a5f0 2021-03-09T10:57:09.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:09.005+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c092380 con 0x7f833800a5f0 2021-03-09T10:57:09.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:09.009+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f833800a5f0 2021-03-09T10:57:09.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:57:09 smithi146 bash[13136]: debug 2021-03-09T10:57:09.004+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 35307 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:57:14.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833802c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:57:14.008 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833802c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:14.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 secure :-1 s=READY pgs=11992 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c06b960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:14.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 secure :-1 s=READY pgs=11992 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c06b960).stop 2021-03-09T10:57:14.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833802c4e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:14.009 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 unknown :-1 s=CLOSED pgs=11992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:14.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:14.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:14.010 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:14.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020580 con 0x7f8324006e60 2021-03-09T10:57:14.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338016a80 con 0x7f833802c100 2021-03-09T10:57:14.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e180 con 0x7f8324006600 2021-03-09T10:57:14.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:57:14.011 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:14.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:57:14.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:14.012 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.005+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:14.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.009+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:14.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.009+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:14.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.009+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:14.013 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.009+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:14.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.009+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024620 con 0x7f833802c100 2021-03-09T10:57:14.014 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.009+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12001 cs=0 l=1 rev1=1 rx=0x7f834402d4d0 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:14.015 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.013+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404af00 con 0x7f833802c100 2021-03-09T10:57:14.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.013+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344075f00 con 0x7f833802c100 2021-03-09T10:57:14.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:14.017+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440496a0 con 0x7f833802c100 2021-03-09T10:57:14.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:57:14 smithi146 bash[13136]: debug 2021-03-09T10:57:14.011+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 35370 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:57:19.016 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:57:19.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:19.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12001 cs=0 l=1 rev1=1 rx=0x7f834402d4d0 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:19.017 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12001 cs=0 l=1 rev1=1 rx=0x7f834402d4d0 tx=0x7f8344007620).stop 2021-03-09T10:57:19.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:19.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=CLOSED pgs=12001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:19.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:19.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:19.018 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:19.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e470 con 0x7f8324006e60 2021-03-09T10:57:19.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e180 con 0x7f833800a5f0 2021-03-09T10:57:19.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338068280 con 0x7f8324006600 2021-03-09T10:57:19.019 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:57:19.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:19.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:57:19.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:19.020 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:19.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:19.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:19.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.013+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:19.021 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.017+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:19.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.017+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338073210 con 0x7f833800a5f0 2021-03-09T10:57:19.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.017+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 secure :-1 s=READY pgs=12009 cs=0 l=1 rev1=1 rx=0x7f833806e2e0 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:19.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.021+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834404ba10 con 0x7f833800a5f0 2021-03-09T10:57:19.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.021+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834405beb0 con 0x7f833800a5f0 2021-03-09T10:57:19.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:19.025+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834400b610 con 0x7f833800a5f0 2021-03-09T10:57:19.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:57:19 smithi146 bash[13136]: debug 2021-03-09T10:57:19.019+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 35430 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:57:24.022 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.017+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833802c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:57:24.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.017+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833802c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:24.023 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.017+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 secure :-1 s=READY pgs=12009 cs=0 l=1 rev1=1 rx=0x7f833806e2e0 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:24.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.017+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 secure :-1 s=READY pgs=12009 cs=0 l=1 rev1=1 rx=0x7f833806e2e0 tx=0x7f8344007620).stop 2021-03-09T10:57:24.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.017+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833802c4e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:24.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.017+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802c4e0 unknown :-1 s=CLOSED pgs=12009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:24.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.017+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:24.024 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.017+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:24.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.021+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:24.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.021+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e470 con 0x7f8324006e60 2021-03-09T10:57:24.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.021+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338025320 con 0x7f833802c100 2021-03-09T10:57:24.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.021+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e180 con 0x7f8324006600 2021-03-09T10:57:24.025 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.021+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:57:24.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.021+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:24.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.021+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:57:24.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.021+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:24.026 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.021+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:24.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.021+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:24.027 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.021+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:24.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.021+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:24.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.021+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:24.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.021+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017b80 con 0x7f833802c100 2021-03-09T10:57:24.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.021+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12018 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:24.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.025+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402cd80 con 0x7f833802c100 2021-03-09T10:57:24.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.025+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f833802c100 2021-03-09T10:57:24.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:24.029+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834404c8f0 con 0x7f833802c100 2021-03-09T10:57:24.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:57:24 smithi146 bash[13136]: debug 2021-03-09T10:57:24.023+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 35493 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:57:29.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2021-03-09T10:57:29.028 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:29.029 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12018 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:29.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12018 cs=0 l=1 rev1=1 rx=0x7f83440076f0 tx=0x7f8344007620).stop 2021-03-09T10:57:29.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:29.030 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=CLOSED pgs=12018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:29.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:29.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:29.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:29.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e180 con 0x7f8324006e60 2021-03-09T10:57:29.031 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806b6f0 con 0x7f833800a5f0 2021-03-09T10:57:29.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f8324006600 2021-03-09T10:57:29.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:57:29.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:29.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:57:29.032 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:29.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:29.033 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:29.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:29.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:29.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:29.034 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338016690 con 0x7f833800a5f0 2021-03-09T10:57:29.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.029+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802cf30 secure :-1 s=READY pgs=12026 cs=0 l=1 rev1=1 rx=0x7f833806d720 tx=0x7f833c06b960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:29.035 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.033+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c079800 con 0x7f833800a5f0 2021-03-09T10:57:29.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.033+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0a1690 con 0x7f833800a5f0 2021-03-09T10:57:29.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:29.037+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f833800a5f0 2021-03-09T10:57:29.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:57:29 smithi146 bash[13136]: debug 2021-03-09T10:57:29.031+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 35553 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:57:34.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833802cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:57:34.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833802cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:34.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802cf30 secure :-1 s=READY pgs=12026 cs=0 l=1 rev1=1 rx=0x7f833806d720 tx=0x7f833c06b960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:34.036 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802cf30 secure :-1 s=READY pgs=12026 cs=0 l=1 rev1=1 rx=0x7f833806d720 tx=0x7f833c06b960).stop 2021-03-09T10:57:34.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f833802cf30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:34.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f833802cf30 unknown :-1 s=CLOSED pgs=12026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:34.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:34.037 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:34.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:34.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e470 con 0x7f8324006e60 2021-03-09T10:57:34.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e180 con 0x7f833802c100 2021-03-09T10:57:34.038 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338029980 con 0x7f8324006600 2021-03-09T10:57:34.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:57:34.039 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:34.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:57:34.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:34.040 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.033+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:34.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.037+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:34.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.037+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:34.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.037+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:34.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.037+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:34.041 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.037+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338071c50 con 0x7f833802c100 2021-03-09T10:57:34.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.037+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12039 cs=0 l=1 rev1=1 rx=0x7f833c012040 tx=0x7f833c06b960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:34.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.041+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa080 con 0x7f833802c100 2021-03-09T10:57:34.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.045+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0254c0 con 0x7f833802c100 2021-03-09T10:57:34.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:34.045+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c029d60 con 0x7f833802c100 2021-03-09T10:57:34.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:57:34 smithi146 bash[13136]: debug 2021-03-09T10:57:34.039+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 35616 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:57:39.042 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.037+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:57:39.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.037+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:39.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.037+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12039 cs=0 l=1 rev1=1 rx=0x7f833c012040 tx=0x7f833c06b960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:39.043 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.037+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12039 cs=0 l=1 rev1=1 rx=0x7f833c012040 tx=0x7f833c06b960).stop 2021-03-09T10:57:39.044 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.037+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:39.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=CLOSED pgs=12039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:39.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:39.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:39.045 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f8338066090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:39.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f8324006e60 2021-03-09T10:57:39.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065820 con 0x7f833800a5f0 2021-03-09T10:57:39.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802e180 con 0x7f8324006600 2021-03-09T10:57:39.046 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:57:39.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:39.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:57:39.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:39.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f8338066090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:39.047 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:39.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:39.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:39.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:39.048 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338024620 con 0x7f833800a5f0 2021-03-09T10:57:39.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.041+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f8338066090 secure :-1 s=READY pgs=12044 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c06b960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:39.049 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.045+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f833800a5f0 2021-03-09T10:57:39.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.049+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0218d0 con 0x7f833800a5f0 2021-03-09T10:57:39.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:39.049+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833800a5f0 2021-03-09T10:57:39.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:57:39 smithi146 bash[13136]: debug 2021-03-09T10:57:39.043+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 35676 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:57:44.050 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f8338066090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:57:44.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f8338066090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:44.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f8338066090 secure :-1 s=READY pgs=12044 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c06b960).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:44.051 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f8338066090 secure :-1 s=READY pgs=12044 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c06b960).stop 2021-03-09T10:57:44.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f8338066090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:44.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f8338066090 unknown :-1 s=CLOSED pgs=12044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:44.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:44.052 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:44.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:44.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806dfa0 con 0x7f8324006e60 2021-03-09T10:57:44.053 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d450 con 0x7f833802c100 2021-03-09T10:57:44.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338069ff0 con 0x7f8324006600 2021-03-09T10:57:44.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:57:44.054 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:44.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:57:44.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:44.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:44.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:44.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:44.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:44.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:44.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017d20 con 0x7f833802c100 2021-03-09T10:57:44.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.049+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12052 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c029fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:44.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.053+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f833802c100 2021-03-09T10:57:44.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.053+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c007640 con 0x7f833802c100 2021-03-09T10:57:44.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:44.057+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0459d0 con 0x7f833802c100 2021-03-09T10:57:44.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:57:44 smithi146 bash[13136]: debug 2021-03-09T10:57:44.051+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 35739 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:57:49.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:57:49.055 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:49.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12052 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c029fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:49.056 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12052 cs=0 l=1 rev1=1 rx=0x7f833c001070 tx=0x7f833c029fd0).stop 2021-03-09T10:57:49.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:49.057 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=CLOSED pgs=12052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:49.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:49.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:49.058 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f83380727c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:49.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806d1e0 con 0x7f8324006e60 2021-03-09T10:57:49.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380205e0 con 0x7f833800a5f0 2021-03-09T10:57:49.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e290 con 0x7f8324006600 2021-03-09T10:57:49.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:57:49.059 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:49.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:57:49.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:49.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f83380727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:49.060 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:49.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:49.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:49.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:49.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022220 con 0x7f833800a5f0 2021-03-09T10:57:49.061 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.053+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f83380727c0 secure :-1 s=READY pgs=12061 cs=0 l=1 rev1=1 rx=0x7f833c092bd0 tx=0x7f833c029fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:49.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.061+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f833800a5f0 2021-03-09T10:57:49.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.061+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c06b730 con 0x7f833800a5f0 2021-03-09T10:57:49.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:49.061+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05cd50 con 0x7f833800a5f0 2021-03-09T10:57:49.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:57:49 smithi146 bash[13136]: debug 2021-03-09T10:57:49.059+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 35799 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:57:54.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f83380727c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:57:54.062 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f83380727c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:54.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f83380727c0 secure :-1 s=READY pgs=12061 cs=0 l=1 rev1=1 rx=0x7f833c092bd0 tx=0x7f833c029fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:54.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f83380727c0 secure :-1 s=READY pgs=12061 cs=0 l=1 rev1=1 rx=0x7f833c092bd0 tx=0x7f833c029fd0).stop 2021-03-09T10:57:54.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 msgr2=0x7f83380727c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:54.063 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800a5f0 0x7f83380727c0 unknown :-1 s=CLOSED pgs=12061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:54.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:54.064 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:54.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:57:54.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:54.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:54.065 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d7c0 con 0x7f8324006e60 2021-03-09T10:57:54.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004b70 con 0x7f833802c100 2021-03-09T10:57:54.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802b0c0 con 0x7f8324006600 2021-03-09T10:57:54.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:57:54.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:54.066 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.060+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:54.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.064+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:54.067 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.064+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:54.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.064+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:54.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.064+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:54.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.064+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833802c100 2021-03-09T10:57:54.068 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.064+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12071 cs=0 l=1 rev1=1 rx=0x7f833c010920 tx=0x7f833c029fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:54.069 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.064+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c010530 con 0x7f833802c100 2021-03-09T10:57:54.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.068+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c010350 con 0x7f833802c100 2021-03-09T10:57:54.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:54.068+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0792c0 con 0x7f833802c100 2021-03-09T10:57:54.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:57:54 smithi146 bash[13136]: debug 2021-03-09T10:57:54.067+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 35862 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:57:59.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.064+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:57:59.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.064+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:59.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.064+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12071 cs=0 l=1 rev1=1 rx=0x7f833c010920 tx=0x7f833c029fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:59.070 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.064+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12071 cs=0 l=1 rev1=1 rx=0x7f833c010920 tx=0x7f833c029fd0).stop 2021-03-09T10:57:59.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.064+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:59.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.064+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=CLOSED pgs=12071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:59.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.064+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:59.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.064+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:59.072 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.064+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833800f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:59.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.064+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338020b20 con 0x7f8324006e60 2021-03-09T10:57:59.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.064+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338065820 con 0x7f833800f4f0 2021-03-09T10:57:59.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.064+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004e60 con 0x7f8324006600 2021-03-09T10:57:59.073 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.064+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:57:59.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.068+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:59.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.068+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:57:59.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.068+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:59.074 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.068+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833800f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:59.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.068+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:59.075 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.068+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.068+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.068+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.068+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833802ea10 con 0x7f833800f4f0 2021-03-09T10:57:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.068+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833800f970 secure :-1 s=READY pgs=12078 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c029fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:59.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.072+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00e260 con 0x7f833800f4f0 2021-03-09T10:57:59.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.072+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c010530 con 0x7f833800f4f0 2021-03-09T10:57:59.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:57:59.076+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0459d0 con 0x7f833800f4f0 2021-03-09T10:57:59.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:57:59 smithi146 bash[13136]: debug 2021-03-09T10:57:59.075+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 35922 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:58:04.077 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f833800f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:58:04.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f833800f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:04.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833800f970 secure :-1 s=READY pgs=12078 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c029fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:04.078 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833800f970 secure :-1 s=READY pgs=12078 cs=0 l=1 rev1=1 rx=0x7f833c04e370 tx=0x7f833c029fd0).stop 2021-03-09T10:58:04.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f833800f970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:04.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833800f970 unknown :-1 s=CLOSED pgs=12078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:04.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:04.081 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:04.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:04.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e290 con 0x7f8324006e60 2021-03-09T10:58:04.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d7c0 con 0x7f833802c100 2021-03-09T10:58:04.082 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338010180 con 0x7f8324006600 2021-03-09T10:58:04.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:58:04.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:58:04.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:04.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:04.083 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.072+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:04.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.076+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:04.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.076+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:04.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.076+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:04.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.076+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:04.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.076+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338022220 con 0x7f833802c100 2021-03-09T10:58:04.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.076+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12087 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c029fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:04.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.080+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00ed00 con 0x7f833802c100 2021-03-09T10:58:04.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.080+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c025e60 con 0x7f833802c100 2021-03-09T10:58:04.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:04.084+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c04d170 con 0x7f833802c100 2021-03-09T10:58:04.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:58:04 smithi146 bash[13136]: debug 2021-03-09T10:58:04.079+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 35985 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:58:09.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:58:09.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:09.084 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12087 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c029fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:09.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 secure :-1 s=READY pgs=12087 cs=0 l=1 rev1=1 rx=0x7f833c0075d0 tx=0x7f833c029fd0).stop 2021-03-09T10:58:09.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338071760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:09.085 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338071760 unknown :-1 s=CLOSED pgs=12087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:09.086 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:09.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833801f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:09.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:09.087 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338021960 con 0x7f8324006e60 2021-03-09T10:58:09.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f833800f4f0 2021-03-09T10:58:09.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338024b90 con 0x7f8324006600 2021-03-09T10:58:09.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:58:09.088 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:09.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:58:09.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:09.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833801f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:09.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:09.089 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:09.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:09.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:09.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020590 con 0x7f833800f4f0 2021-03-09T10:58:09.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.084+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833801f9e0 secure :-1 s=READY pgs=12097 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:09.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f833800f4f0 2021-03-09T10:58:09.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f833800f4f0 2021-03-09T10:58:09.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:09.092+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440171c0 con 0x7f833800f4f0 2021-03-09T10:58:09.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:58:09 smithi146 bash[13136]: debug 2021-03-09T10:58:09.087+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 36045 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:58:14.090 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.084+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f833801f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:58:14.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.084+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f833801f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:14.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.084+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833801f9e0 secure :-1 s=READY pgs=12097 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:14.091 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.084+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833801f9e0 secure :-1 s=READY pgs=12097 cs=0 l=1 rev1=1 rx=0x7f834406efd0 tx=0x7f8344007620).stop 2021-03-09T10:58:14.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f833801f9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:14.092 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.088+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833801f9e0 unknown :-1 s=CLOSED pgs=12097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:14.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.088+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:14.093 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.088+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:14.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.088+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338021080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:14.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338026a90 con 0x7f8324006e60 2021-03-09T10:58:14.094 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f833802c100 2021-03-09T10:58:14.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.088+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004e60 con 0x7f8324006600 2021-03-09T10:58:14.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.088+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:58:14.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.088+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:14.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.088+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:58:14.095 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.088+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:14.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.088+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:14.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.092+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:14.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.092+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:14.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.092+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:14.096 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.092+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:14.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.092+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017d20 con 0x7f833802c100 2021-03-09T10:58:14.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.092+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338021080 secure :-1 s=READY pgs=12107 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c029fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:14.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.096+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c099ee0 con 0x7f833802c100 2021-03-09T10:58:14.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.096+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c099ee0 con 0x7f833802c100 2021-03-09T10:58:14.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:14.096+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f833802c100 2021-03-09T10:58:14.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:58:14 smithi146 bash[13136]: debug 2021-03-09T10:58:14.095+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 36108 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:58:19.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:58:19.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:19.097 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338021080 secure :-1 s=READY pgs=12107 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c029fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:19.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338021080 secure :-1 s=READY pgs=12107 cs=0 l=1 rev1=1 rx=0x7f833c009390 tx=0x7f833c029fd0).stop 2021-03-09T10:58:19.098 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f8338021080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:19.099 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f8338021080 unknown :-1 s=CLOSED pgs=12107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:19.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:19.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833801f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:19.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:58:19.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:19.100 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:19.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806e290 con 0x7f8324006e60 2021-03-09T10:58:19.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338029980 con 0x7f833800f4f0 2021-03-09T10:58:19.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e950 con 0x7f8324006600 2021-03-09T10:58:19.101 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:58:19.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:19.102 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833801f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:19.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:19.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:19.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:19.103 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f833805e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:19.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.096+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278f0 con 0x7f833800f4f0 2021-03-09T10:58:19.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.100+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833801f9e0 secure :-1 s=READY pgs=12117 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344007620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:19.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.100+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834402b860 con 0x7f833800f4f0 2021-03-09T10:58:19.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.104+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834402b860 con 0x7f833800f4f0 2021-03-09T10:58:19.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:19.104+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f8344047070 con 0x7f833800f4f0 2021-03-09T10:58:19.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:58:19 smithi146 bash[13136]: debug 2021-03-09T10:58:19.098+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 36167 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:58:24.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f833801f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:58:24.104 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f833801f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:24.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833801f9e0 secure :-1 s=READY pgs=12117 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344007620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:24.105 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833801f9e0 secure :-1 s=READY pgs=12117 cs=0 l=1 rev1=1 rx=0x7f834404c3c0 tx=0x7f8344007620).stop 2021-03-09T10:58:24.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f833801f9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:24.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f833801f9e0 unknown :-1 s=CLOSED pgs=12117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:24.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338009560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:24.106 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:24.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:24.107 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338022270 con 0x7f8324006e60 2021-03-09T10:58:24.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338063420 con 0x7f833802c100 2021-03-09T10:58:24.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004e60 con 0x7f8324006600 2021-03-09T10:58:24.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338009560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:58:24.108 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338009560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:24.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:58:24.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:24.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:24.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:24.109 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:24.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f8338009560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:24.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f8338009560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:24.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833806efe0 con 0x7f833802c100 2021-03-09T10:58:24.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.104+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 secure :-1 s=READY pgs=12122 cs=0 l=1 rev1=1 rx=0x7f833c0a2590 tx=0x7f833c029fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:24.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.108+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f833802c100 2021-03-09T10:58:24.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.108+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079630 con 0x7f833802c100 2021-03-09T10:58:24.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:24.108+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0792c0 con 0x7f833802c100 2021-03-09T10:58:24.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:58:24 smithi146 bash[13136]: debug 2021-03-09T10:58:24.106+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 36230 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:58:29.110 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:58:29.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:29.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 secure :-1 s=READY pgs=12122 cs=0 l=1 rev1=1 rx=0x7f833c0a2590 tx=0x7f833c029fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:29.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 secure :-1 s=READY pgs=12122 cs=0 l=1 rev1=1 rx=0x7f833c0a2590 tx=0x7f833c029fd0).stop 2021-03-09T10:58:29.111 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:29.112 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=12122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:29.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:29.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:29.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:29.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:58:29.113 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:29.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:58:29.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:29.114 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d7c0 con 0x7f8324006e60 2021-03-09T10:58:29.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004e60 con 0x7f833800f4f0 2021-03-09T10:58:29.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802a6b0 con 0x7f8324006600 2021-03-09T10:58:29.115 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.108+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:29.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.112+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:29.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.112+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:29.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.112+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:29.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.112+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:29.116 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.112+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278f0 con 0x7f833800f4f0 2021-03-09T10:58:29.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.112+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 secure :-1 s=READY pgs=12132 cs=0 l=1 rev1=1 rx=0x7f833c0a8650 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:29.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.120+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa030 con 0x7f833800f4f0 2021-03-09T10:58:29.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.120+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0a8490 con 0x7f833800f4f0 2021-03-09T10:58:29.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:29.124+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c061300 con 0x7f833800f4f0 2021-03-09T10:58:29.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:58:29 smithi146 bash[13136]: debug 2021-03-09T10:58:29.114+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 36290 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:58:34.117 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:58:34.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:34.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 secure :-1 s=READY pgs=12132 cs=0 l=1 rev1=1 rx=0x7f833c0a8650 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:34.118 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 secure :-1 s=READY pgs=12132 cs=0 l=1 rev1=1 rx=0x7f833c0a8650 tx=0x7f833c04de90).stop 2021-03-09T10:58:34.119 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:34.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 unknown :-1 s=CLOSED pgs=12132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:34.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:34.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:34.120 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:34.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e950 con 0x7f8324006e60 2021-03-09T10:58:34.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806b6f0 con 0x7f833802c100 2021-03-09T10:58:34.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833800d7c0 con 0x7f8324006600 2021-03-09T10:58:34.121 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:58:34.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:34.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:58:34.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:34.122 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:34.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:34.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:34.123 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:34.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:34.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338020590 con 0x7f833802c100 2021-03-09T10:58:34.124 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.115+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 secure :-1 s=READY pgs=12139 cs=0 l=1 rev1=1 rx=0x7f833c0072e0 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:34.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.119+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0a8960 con 0x7f833802c100 2021-03-09T10:58:34.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.119+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0a8960 con 0x7f833802c100 2021-03-09T10:58:34.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:34.123+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c05c380 con 0x7f833802c100 2021-03-09T10:58:34.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:58:34 smithi146 bash[13136]: debug 2021-03-09T10:58:34.122+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 36352 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:58:39.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:58:39.125 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:39.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 secure :-1 s=READY pgs=12139 cs=0 l=1 rev1=1 rx=0x7f833c0072e0 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:39.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 secure :-1 s=READY pgs=12139 cs=0 l=1 rev1=1 rx=0x7f833c0072e0 tx=0x7f833c04de90).stop 2021-03-09T10:58:39.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:39.126 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=12139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:39.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:39.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:39.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:39.127 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833806b8e0 con 0x7f8324006e60 2021-03-09T10:58:39.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f833800f4f0 2021-03-09T10:58:39.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b4a0 con 0x7f8324006600 2021-03-09T10:58:39.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:58:39.128 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:39.129 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:58:39.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:39.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.123+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:39.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.127+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:39.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.127+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:39.130 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.127+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:39.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.127+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:39.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.127+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800e310 con 0x7f833800f4f0 2021-03-09T10:58:39.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.127+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 secure :-1 s=READY pgs=12148 cs=0 l=1 rev1=1 rx=0x7f834403cf50 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:39.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.131+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f833800f4f0 2021-03-09T10:58:39.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.131+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f833800f4f0 2021-03-09T10:58:39.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:39.135+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f83440470e0 con 0x7f833800f4f0 2021-03-09T10:58:39.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:58:39 smithi146 bash[13136]: debug 2021-03-09T10:58:39.126+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 36413 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:58:44.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:58:44.131 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:44.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 secure :-1 s=READY pgs=12148 cs=0 l=1 rev1=1 rx=0x7f834403cf50 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:44.132 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 secure :-1 s=READY pgs=12148 cs=0 l=1 rev1=1 rx=0x7f834403cf50 tx=0x7f8344004650).stop 2021-03-09T10:58:44.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:44.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 unknown :-1 s=CLOSED pgs=12148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:44.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:44.133 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:44.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:44.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380662d0 con 0x7f8324006e60 2021-03-09T10:58:44.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801f390 con 0x7f833802c100 2021-03-09T10:58:44.134 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004e60 con 0x7f8324006600 2021-03-09T10:58:44.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:58:44.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:44.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:58:44.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:44.135 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:44.136 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:44.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:44.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:44.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:44.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338017d20 con 0x7f833802c100 2021-03-09T10:58:44.137 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.131+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 secure :-1 s=READY pgs=12156 cs=0 l=1 rev1=1 rx=0x7f8344011040 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:44.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.139+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f833802c100 2021-03-09T10:58:44.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.139+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f8344017640 con 0x7f833802c100 2021-03-09T10:58:44.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:44.139+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ac90 con 0x7f833802c100 2021-03-09T10:58:44.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:58:44 smithi146 bash[13136]: debug 2021-03-09T10:58:44.134+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 36475 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:58:49.138 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.135+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:58:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.135+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.135+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 secure :-1 s=READY pgs=12156 cs=0 l=1 rev1=1 rx=0x7f8344011040 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.135+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 secure :-1 s=READY pgs=12156 cs=0 l=1 rev1=1 rx=0x7f8344011040 tx=0x7f8344004650).stop 2021-03-09T10:58:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.135+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.135+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=12156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:49.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.135+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:49.140 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.135+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:49.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.139+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:49.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.139+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e000 con 0x7f8324006e60 2021-03-09T10:58:49.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.139+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338029980 con 0x7f833800f4f0 2021-03-09T10:58:49.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.139+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f8324006600 2021-03-09T10:58:49.141 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.139+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:58:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.139+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.139+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:58:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.139+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.139+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:49.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.143+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:49.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.143+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:49.143 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.143+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:49.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.143+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:49.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.143+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f83380278f0 con 0x7f833800f4f0 2021-03-09T10:58:49.144 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.143+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 secure :-1 s=READY pgs=12168 cs=0 l=1 rev1=1 rx=0x7f834403bf80 tx=0x7f8344004650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:49.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.147+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834407b1d0 con 0x7f833800f4f0 2021-03-09T10:58:49.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.151+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f834400de80 con 0x7f833800f4f0 2021-03-09T10:58:49.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:49.151+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f834402ac90 con 0x7f833800f4f0 2021-03-09T10:58:49.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:58:49 smithi146 bash[13136]: debug 2021-03-09T10:58:49.142+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 36536 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:58:54.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.143+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:58:54.147 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.143+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f8338025e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:54.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.143+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 secure :-1 s=READY pgs=12168 cs=0 l=1 rev1=1 rx=0x7f834403bf80 tx=0x7f8344004650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:54.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.143+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 secure :-1 s=READY pgs=12168 cs=0 l=1 rev1=1 rx=0x7f834403bf80 tx=0x7f8344004650).stop 2021-03-09T10:58:54.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 msgr2=0x7f8338025e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:54.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833800f4f0 0x7f8338025e80 unknown :-1 s=CLOSED pgs=12168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:54.148 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:54.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:54.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:54.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801b4a0 con 0x7f8324006e60 2021-03-09T10:58:54.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380719a0 con 0x7f833802c100 2021-03-09T10:58:54.149 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d890 con 0x7f8324006600 2021-03-09T10:58:54.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:58:54.150 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:54.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:58:54.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:54.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:54.151 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068240 con 0x7f833802c100 2021-03-09T10:58:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.147+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 secure :-1 s=READY pgs=12174 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:54.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.151+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c079800 con 0x7f833802c100 2021-03-09T10:58:54.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.151+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c079800 con 0x7f833802c100 2021-03-09T10:58:54.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:54.155+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c0352a0 con 0x7f833802c100 2021-03-09T10:58:54.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:58:54 smithi146 bash[13136]: debug 2021-03-09T10:58:54.150+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 36597 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:58:59.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:58:59.153 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:59.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 secure :-1 s=READY pgs=12174 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:59.154 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 secure :-1 s=READY pgs=12174 cs=0 l=1 rev1=1 rx=0x7f833c04e6e0 tx=0x7f833c04de90).stop 2021-03-09T10:58:59.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:59.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=12174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:59.155 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:59.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:59.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 0x7f833800a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:59.156 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012880 con 0x7f8324006e60 2021-03-09T10:58:59.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004e60 con 0x7f8338009f80 2021-03-09T10:58:59.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833801e000 con 0x7f8324006600 2021-03-09T10:58:59.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:58:59.157 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:59.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:58:59.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:59.158 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.151+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 0x7f833800a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:59.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.155+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:59.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.155+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:59.159 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.155+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:59.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.155+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:59.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.155+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833800a8f0 con 0x7f8338009f80 2021-03-09T10:58:59.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.155+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 0x7f833800a400 secure :-1 s=READY pgs=12182 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:59.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.159+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0578b0 con 0x7f8338009f80 2021-03-09T10:58:59.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.159+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0553a0 con 0x7f8338009f80 2021-03-09T10:58:59.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:58:59.163+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c055190 con 0x7f8338009f80 2021-03-09T10:58:59.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:58:59 smithi146 bash[13136]: debug 2021-03-09T10:58:59.154+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 36659 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:59:04.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.159+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 msgr2=0x7f833800a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:59:04.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.159+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 msgr2=0x7f833800a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:59:04.160 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.159+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 0x7f833800a400 secure :-1 s=READY pgs=12182 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:59:04.161 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.159+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 0x7f833800a400 secure :-1 s=READY pgs=12182 cs=0 l=1 rev1=1 rx=0x7f833c000c00 tx=0x7f833c04de90).stop 2021-03-09T10:59:04.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.159+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 msgr2=0x7f833800a400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:59:04.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.159+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 0x7f833800a400 unknown :-1 s=CLOSED pgs=12182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:59:04.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.159+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:04.162 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.159+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:04.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.159+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:04.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.159+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802b600 con 0x7f8324006e60 2021-03-09T10:59:04.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.159+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338008c20 con 0x7f833802c100 2021-03-09T10:59:04.163 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.159+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338074fe0 con 0x7f8324006600 2021-03-09T10:59:04.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.159+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:59:04.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.159+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:04.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.159+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:59:04.164 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.159+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:04.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.163+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:59:04.165 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.163+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:04.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.163+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:04.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.163+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:04.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.163+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:04.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.163+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338068240 con 0x7f833802c100 2021-03-09T10:59:04.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.163+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 secure :-1 s=READY pgs=12195 cs=0 l=1 rev1=1 rx=0x7f833c01d040 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:59:04.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.167+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c007210 con 0x7f833802c100 2021-03-09T10:59:04.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.171+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0414c0 con 0x7f833802c100 2021-03-09T10:59:04.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:04.171+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f833802c100 2021-03-09T10:59:04.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:59:04 smithi146 bash[13136]: debug 2021-03-09T10:59:04.162+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 36720 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:59:09.166 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.163+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:59:09.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.163+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f833805e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:59:09.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.163+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 secure :-1 s=READY pgs=12195 cs=0 l=1 rev1=1 rx=0x7f833c01d040 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:59:09.167 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.163+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 secure :-1 s=READY pgs=12195 cs=0 l=1 rev1=1 rx=0x7f833c01d040 tx=0x7f833c04de90).stop 2021-03-09T10:59:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.163+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 msgr2=0x7f833805e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:59:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.163+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=CLOSED pgs=12195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:59:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.163+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.163+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.163+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 0x7f833800d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.163+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338064450 con 0x7f8324006e60 2021-03-09T10:59:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.163+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833802d890 con 0x7f8338009f80 2021-03-09T10:59:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.163+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338004e60 con 0x7f8324006600 2021-03-09T10:59:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.163+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:59:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.163+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.163+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:59:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.167+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.167+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 0x7f833800d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:59:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.167+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.167+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.167+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.167+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.167+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f8338002850 con 0x7f8338009f80 2021-03-09T10:59:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.167+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 0x7f833800d970 secure :-1 s=READY pgs=12200 cs=0 l=1 rev1=1 rx=0x7f833c092050 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:59:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.171+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c010960 con 0x7f8338009f80 2021-03-09T10:59:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.171+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c0414c0 con 0x7f8338009f80 2021-03-09T10:59:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:09.175+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c057500 con 0x7f8338009f80 2021-03-09T10:59:09.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:59:09 smithi146 bash[13136]: debug 2021-03-09T10:59:09.170+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 36782 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:59:14.173 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 msgr2=0x7f833800d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T10:59:14.174 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 msgr2=0x7f833800d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:59:14.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 0x7f833800d970 secure :-1 s=READY pgs=12200 cs=0 l=1 rev1=1 rx=0x7f833c092050 tx=0x7f833c04de90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:59:14.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 0x7f833800d970 secure :-1 s=READY pgs=12200 cs=0 l=1 rev1=1 rx=0x7f833c092050 tx=0x7f833c04de90).stop 2021-03-09T10:59:14.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 msgr2=0x7f833800d970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:59:14.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8338009f80 0x7f833800d970 unknown :-1 s=CLOSED pgs=12200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:59:14.175 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:14.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:14.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f8343fff700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:14.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801eb50 con 0x7f8324006e60 2021-03-09T10:59:14.176 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833801d080 con 0x7f833802c100 2021-03-09T10:59:14.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8338012880 con 0x7f8324006600 2021-03-09T10:59:14.177 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f834b7fe700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2021-03-09T10:59:14.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f834b7fe700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:14.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2021-03-09T10:59:14.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:14.178 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.170+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:59:14.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.174+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 msgr2=0x7f8338006e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:14.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.174+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8324006600 0x7f8338006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:14.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.174+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 msgr2=0x7f83380200a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:14.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.174+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8324006e60 0x7f83380200a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:14.179 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.174+0000 7f834affd700 1 -- 172.21.15.5:0/3543657960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=4+}) v3 -- 0x7f833801e950 con 0x7f833802c100 2021-03-09T10:59:14.180 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.174+0000 7f834affd700 1 --2- 172.21.15.5:0/3543657960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f833802c100 0x7f833805e5b0 secure :-1 s=READY pgs=12208 cs=0 l=1 rev1=1 rx=0x7f833c092900 tx=0x7f833c04de90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:59:14.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.178+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0aa070 con 0x7f833802c100 2021-03-09T10:59:14.181 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.178+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1032+0+0 (secure 0 0 0) 0x7f833c092740 con 0x7f833802c100 2021-03-09T10:59:14.183 INFO:teuthology.orchestra.run.smithi005.stderr:2021-03-09T10:59:14.182+0000 7f8343fff700 1 -- 172.21.15.5:0/3543657960 <== mon.2 v2:172.21.15.146:3300/0 3 ==== mgrmap(e 14) v1 ==== 71873+0+0 (secure 0 0 0) 0x7f833c007000 con 0x7f833802c100 2021-03-09T10:59:14.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Mar 09 10:59:14 smithi146 bash[13136]: debug 2021-03-09T10:59:14.174+0000 7fa560337700 -1 mon.b@2(probing) e3 get_health_metrics reporting 36843 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T08:59:59.937663+0000) 2021-03-09T10:59:15.982 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-03-09T10:59:15.984 DEBUG:teuthology.task.console_log:Killing console logger for smithi005 2021-03-09T10:59:15.984 DEBUG:teuthology.task.console_log:Killing console logger for smithi146 2021-03-09T10:59:15.985 DEBUG:teuthology.exit:Finished running handlers